Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
In short
An embedding is a fixed-length list of numbers that stands for the meaning of a text, so texts with similar meaning get lists that point in similar directions. Search, RAG and clustering compare these lists instead of comparing words.
In short
An embedding is a fixed-length list of numbers that stands for the meaning of a text, so texts with similar meaning get lists that point in similar directions. Search, RAG and clustering compare these lists instead of comparing words.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Embedding across a run of lessons, not one page. These 4 alone are about 245 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
An embedding turns a piece of text into a list of numbers, so that texts with similar meaning get similar numbers. This lesson measures two real embedding models on this laptop. With the English-trained model, sentences with the same meaning scored 0.89 to 0.96 and unrelated ones 0.52 to 0.60. Then the test that matters for a global audience: the same sentence in six other languages. The English-trained model scored five of the six translations below an unrelated English sentence. A multilingual model scored all six 0.92 to 0.98.
Ollama can return only the first N numbers of an embedding. Measured on 470 quiz questions and 1,823 pieces of this course: nomic-embed-text found the right lesson in the top 5 for 431 questions at 256 of its 768 numbers, against 441 at full size. bge-m3 found 414 at 256 of 1,024, against 439, and fell faster below that. Keeping the first numbers instead of the last helped nomic only a little. At 16 numbers, the small example picked the wrong text.
Every customer of this shop bought products with names like 'regency cakestand 3 tier'. I turned all 5,257 names into lists of numbers twice, once with a local neural model and once with word counts, averaged them per customer, and added them to lesson 1's six features. Over 20 seeds and 26,851 real test rows, no embedding gave a gain I could tell apart from luck. A control built from random lists, which knows which products a customer bought but nothing about their names, did as well as either embedding; how far ahead or behind it looks depends on which random table you draw. The vectors did group similar products; the model did not need that.
A hands-on lab on two real models. An easy test set cannot tell a free model from a paid one, 30 near misses do more damage than 2,000 unrelated documents, and a missing prefix costs 20 points of recall.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with What an Embedding Is: Meaning as a List of Numberspay once,
yours for life
A computer cannot compare meanings. It can only compare numbers. An embedding turns a piece of text into a list of numbers, also called a vector, so that texts with similar meaning get similar lists. "How do I reset my password?" and "I forgot my password, how can I change it?" share few words, but their embeddings sit close together.
Picture a huge map where every sentence gets a pin. Sentences that mean the same thing get pins close together, and unrelated sentences get pins far apart. An embedding is the position of that pin, written as numbers. The only difference is that this map has hundreds of directions instead of two.
Tokens are different. A token ID is only a label: in GPT-4o's tokenizer, ID 382 is the word " is", and the ID next to it is not any closer in meaning. In an embedding, distance means something. That is what lets a program match meaning instead of spelling, using ordinary arithmetic.
An embedding model is a language model trained for this one job. It cuts the text into tokens, reads them together, and returns one list of numbers for the whole text. The length of the list is called the number of dimensions, and it is fixed for each model: nomic-embed-text returns 768 numbers, bge-m3 returns 1,024, and OpenAI's text-embedding-3-small returns 1,536.
During training, the model sees many pairs of texts that belong together, such as a question and its answer, and many pairs that do not. It is adjusted step by step, so that pairs that belong together point the same way and other pairs point apart. Nobody tells it what each number means, so no single number has a meaning you can name. The meaning is spread across the whole list.
The same text gives the same list from the same model, or very nearly the same on different hardware. So you embed your documents once, store the lists, and at question time you embed only the question and compare. The usual comparison is cosine similarity, a score based on the angle between two lists, where a value near 1 means very close in meaning.
In a measured lab run with nomic-embed-text, nine sentence pairs written in advance fell into three clear groups. Pairs with the same meaning scored 0.89 to 0.96. Pairs on the same topic scored 0.71 to 0.76. Unrelated pairs scored 0.52 to 0.60. The groups did not overlap, and that separation is what makes search work.
Notice that unrelated sentences scored above 0.5, not near 0. A score only means something next to other scores from the same model. A fixed rule such as "above 0.5 is a match" fails: an unrelated sentence about bread scored 0.578 with nomic-embed-text, above the line, and 0.429 with bge-m3, below it.
The language test mattered most. One English sentence was translated into six languages and each translation was scored against the original. With nomic-embed-text, the translations scored 0.47 to 0.70, and five of the six scored below the unrelated bread sentence; Hindi scored 0.537 against 0.578 for bread. With bge-m3, a multilingual model, every translation scored 0.92 to 0.98, and the bread sentence 0.43. Both models returned numbers for Hindi with no warning. Only the test showed that one set was not useful.
Never mix models. Every document and every question must be embedded by the same model, because each model draws its own map. Comparing a 768-number list with a 1,024-number list in Python ran with no error, because the zip function silently dropped the extra 256 numbers, and it gave a score of -0.007 that meant nothing. If you change models, embed every document again, and keep the source text so that you can.
Read the model's instructions. nomic-embed-text expects "search_query: " in front of a question and "search_document: " in front of a document. The same question embedded with and without its label scored only 0.96 against itself, so leaving the label out changes results. Every embedding model also has a limit on how many tokens it reads, and text past that limit can be cut off without an error.
Storage is simple arithmetic. Stored as 4-byte numbers, one nomic-embed-text embedding takes 768 x 4 = 3,072 bytes, and one bge-m3 embedding takes 1,024 x 4 = 4,096 bytes. For one million paragraphs, that is about 3.1 GB against 4.1 GB, before any index. The models differ in size too: bge-m3 is a 1.2 GB download, nomic-embed-text 274 MB.
Use embeddings when the same meaning can be written in many ways: help-page search, answering questions from your own documents, grouping similar support tickets. If your users write in more than one language, choose a multilingual model, and test it on their real text before you commit.
Do not rely on embeddings when exact strings matter. An order number, an error code or a person's name is a job for keyword matching. Two order numbers that differ by one digit are two different orders, but an embedding model may place them almost on top of each other. Many real systems run both kinds of search.
A single word is a weak input, because there is no sentence around it: "table" can mean a database table or a kitchen table. Full sentences and short paragraphs give the model more to work with. Before you trust a model, write ten or so pairs from your own product, some with the same meaning, some on the same topic and some unrelated, and check that the scores come out in that order with a clear gap.
nomic-embed-text
An open embedding model, available in Ollama, that returns 768 numbers per text and expects search_query and search_document prefixes.
bge-m3
A multilingual embedding model from BAAI that returns 1,024 numbers per text and supports more than 100 languages.
OpenAI text-embedding-3-small
A hosted embedding model from OpenAI that returns 1,536 numbers per text by default.
sentence-transformers
An open-source Python library for computing sentence and paragraph embeddings with many pretrained models.
A token is a piece of text with an ID number, and the ID is only a label with no meaning in its value. An embedding is a list of numbers made for a whole text, placed so that similar meanings sit close together. Tokens are what a model reads; embeddings are what you compare.
No. Each model draws its own map, so a score between them means nothing. In one test, comparing a 768-number and a 1,024-number embedding ran with no error and gave -0.007. Embed documents and questions with the same model, and check that the list lengths match.
There is no fixed number. With one model, unrelated sentences scored 0.52 to 0.60; with another, 0.43 to 0.48. Pick a cut-off for one model by scoring pairs whose answer you already know, and choose again if you change models.
The model decides it: 768, 1,024 and 1,536 are common. More numbers cost more storage, roughly dimensions x 4 bytes per text at full precision. Choose the model by testing it on your own text and languages, then accept its size.
Only if the model was trained on them. In one test, an English-trained model scored five of six translations below an unrelated English sentence, while a multilingual model scored all six between 0.92 and 0.98. Test your users' languages before you choose.
Embedding Storage
How to efficiently store, manage, and serve high-dimensional embedding vectors at scale
Shorter Embeddings: Keep the First Numbers, Measure What You Lose
Ollama can return only the first N numbers of an embedding. Measured on 470 quiz questions and 1,823 pieces of this course: nomic-embed-text found the right lesson in the top 5 for 431 questions at 256 of its 768 numbers, against 441 at full size. bge-m3 found 414 at 256 of 1,024, against 439, and fell faster below that. Keeping the first numbers instead of the last helped nomic only a little. At 16 numbers, the small example picked the wrong text.
Embeddings as Features: Did the Meaning of Product Names Help?
Every customer of this shop bought products with names like 'regency cakestand 3 tier'. I turned all 5,257 names into lists of numbers twice, once with a local neural model and once with word counts, averaged them per customer, and added them to lesson 1's six features. Over 20 seeds and 26,851 real test rows, no embedding gave a gain I could tell apart from luck. A control built from random lists, which knows which products a customer bought but nothing about their names, did as well as either embedding; how far ahead or behind it looks depends on which random table you draw. The vectors did group similar products; the model did not need that.
Choosing an Embedding Model: Why the Leaderboard Is Measuring Someone Else's Documents
A hands-on lab on two real models. An easy test set cannot tell a free model from a paid one, 30 near misses do more damage than 2,000 unrelated documents, and a missing prefix costs 20 points of recall.
The Silent Cut: How Much of Your Text an Embedding Model Really Reads
Send an embedding model a long text and it may read only the start, with no error. Measured with Ollama on this laptop: nomic-embed-text and bge-m3 both stopped at 2,048 tokens by default, although bge-m3's own page says 8,192. A sentence hidden past that point changed the score by exactly 0.000. Raising two settings let bge-m3 read 8,192 tokens, and then the hidden sentence was read but barely moved the score: 0.332 against 0.328.
Vector Database
A database that stores embeddings, lists of numbers that capture the meaning of text or images, and finds the ones closest to a question's embedding. It powers semantic search and RAG.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
Cosine Similarity
A measure of how close two embeddings are, based on the angle between them rather than their length. The standard way to rank how related two pieces of text are in vector search.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
HNSW
Hierarchical Navigable Small World, a graph-based nearest neighbor index that gives high recall at low latency by navigating layers of links, at the cost of more memory than IVF.
This page explains the idea. The "What an Embedding Is: Meaning as a List of Numbers" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the What an Embedding Is: Meaning as a List of Numbers lessoncourse 2
Take models from notebook to production, with labs on real models.