Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Foundation · LLM and GenAI Ops
Embeddings trained so their first numbers carry the most meaning, which lets you cut them shorter to save storage and search time while losing little accuracy.
In short
Matryoshka embeddings are embeddings from a model trained so that the first numbers in the list carry the most meaning, which lets you keep only the front part and still search well. You store less and compute less, and you lose some accuracy that you should measure.
In short
Matryoshka embeddings are embeddings from a model trained so that the first numbers in the list carry the most meaning, which lets you keep only the front part and still search well. You store less and compute less, and you lose some accuracy that you should measure.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Matryoshka Embeddings across a run of lessons, not one page. These 4 alone are about 207 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Ollama can return only the first N numbers of an embedding. Measured on 470 quiz questions and 1,823 pieces of this course: nomic-embed-text found the right lesson in the top 5 for 431 questions at 256 of its 768 numbers, against 441 at full size. bge-m3 found 414 at 256 of 1,024, against 439, and fell faster below that. Keeping the first numbers instead of the last helped nomic only a little. At 16 numbers, the small example picked the wrong text.
An embedding turns a piece of text into a list of numbers, so that texts with similar meaning get similar numbers. This lesson measures two real embedding models on this laptop. With the English-trained model, sentences with the same meaning scored 0.89 to 0.96 and unrelated ones 0.52 to 0.60. Then the test that matters for a global audience: the same sentence in six other languages. The English-trained model scored five of the six translations below an unrelated English sentence. A multilingual model scored all six 0.92 to 0.98.
Every customer of this shop bought products with names like 'regency cakestand 3 tier'. I turned all 5,257 names into lists of numbers twice, once with a local neural model and once with word counts, averaged them per customer, and added them to lesson 1's six features. Over 20 seeds and 26,851 real test rows, no embedding gave a gain I could tell apart from luck. A control built from random lists, which knows which products a customer bought but nothing about their names, did as well as either embedding; how far ahead or behind it looks depends on which random table you draw. The vectors did group similar products; the model did not need that.
Should you embed a whole document as one list of numbers, or cut it into small pieces first? Measured on this course's 112 AI lessons with 470 of their own quiz questions and bge-m3 reading up to 8,192 tokens. With one embedding per lesson, the right lesson came first for 286 questions. With 200-word pieces, for 356. Pieces won on short lessons too, not only long ones, and whole lessons still won 31 questions.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
Open a set of Russian nesting dolls, called matryoshka dolls. The smallest one is tiny, but it is still a whole doll. Nothing important was left off to make it small.
An embedding is a long list of numbers that stands for the meaning of a piece of text. Each number is called a dimension. nomic-embed-text gives 768 numbers per text. bge-m3 gives 1,024.
A Matryoshka embedding is built like the dolls. The model was trained so that its first numbers work on their own as a smaller embedding. The later numbers add finer detail. So you can keep the first 256 numbers, or the first 64, and still have a usable embedding.
The name comes from the training method, Matryoshka Representation Learning, often shortened to MRL.
An embedding is stored for every document, or every piece of a document, that you want to search. Each number usually takes 4 bytes. So the length of the vector is multiplied by how many documents you keep.
For one million documents, the lesson does the arithmetic. At 768 numbers that is about 3.1 GB, just for the embeddings. At 1,024 numbers it is about 4.1 GB. At 256 numbers it is about 1.0 GB, for either model.
Search work shrinks the same way. A simple search compares the question with every stored vector, number by number. In the lesson's test of 1,823 pieces, one question needs 1,400,064 multiplications at 768 numbers and 466,688 at 256.
That saving is only worth having if search still finds the right documents. A smaller vector that finds the wrong document is not cheaper. It is broken. So the real question is how much accuracy you lose.
Ollama, a free program that runs models on your own computer, has a setting called dimensions in its embed endpoint. The lesson checked what it does. It asked for the full vectors, cut them by hand, and compared the result with what Ollama returned for 256 dimensions. They matched to within about one ten-millionth, for both models, on 50 documents.
So the setting does two simple things:
Nothing is recomputed. The setting works on any model, even one never trained for it. That is why the real question is how much meaning sits in the first numbers.
The second step matters. With length 1, cosine similarity and the dot product give the same ranking. Without it, each cut vector has a different length, and length leaks into the ranking. The cosine, dot and distance lesson measured that problem. If you cut vectors yourself, you must rescale too.
In ordinary training, a model is scored on how well its full vector puts texts with the same meaning close together. Then it is adjusted to do better.
In Matryoshka training, each example is scored several times: once with the full vector, again with only its first part, and again with a smaller first part. The model is adjusted to do well on all of them at once. The only way to do well with a short first part is to put the most useful information there. So the model learns to pack meaning at the front.
The lesson is careful here. This explanation is general knowledge about the method, from the MRL paper, not something the lab measured. A model card is the page its makers publish with it. The card for nomic-embed-text v1.5 says in its title that it uses Matryoshka Representation Learning. bge-m3's card says nothing about being cut. That does not prove how bge-m3 was trained, only that its makers do not advertise it.
The lesson also checked which nomic model it tested. In Ollama, the v1.5 tag and the latest tag had the same ID, so the tested model was v1.5.
The test used 112 lessons of this course, cut into 1,823 pieces of 200 words, and the 470 quiz questions inside those lessons. The right answer to each question was the lesson it came from. Each vector was cut to its first 32, 64, 128, 256 or 512 numbers, or kept whole. The measure: was the right lesson in the top 5 results?
The stricter count, right lesson ranked first, shows a twist. At full size, bge-m3 was better, 356 to 339. Cut both to 128 numbers and nomic was well ahead, 297 to 254. Which model is better depends on how short you need the vectors to be.
Was the meaning really at the front? The lesson kept the last N numbers instead of the first and compared. For nomic, the first numbers won by 20 at 64 numbers and by 5 to 7 at 128 and 256. The training works, but the effect was small. For bge-m3, first and last made almost no difference. And nomic's last 64 numbers, 361, still beat bge-m3's first 64, 315.
A small example showed the failure mode clearly. With five short texts and one question about too many requests, 16 numbers picked the text about sharding. Shorter vectors make texts look more alike, so all scores rise. Judge a size by which document wins, never by how high the scores are.
Put the cost next to the loss. For nomic, going from 768 to 256 numbers saved two thirds of the space and cost 10 of 441 right answers, about 2 percent. Going to 64 numbers cost 60, about 14 percent. Whether either trade is worth it depends on your data and your budget.
Do not cut at all when you have few documents, because the saving is too small. And do not cut when you cannot measure. Without known answers, you cannot see what the cut costs.
Keeping fewer numbers is one way to shrink vectors. The other is quantization: keep every number, but store each one with less detail. The lesson did not test the two together.
The Shorter Embeddings lesson has the runnable script, the full results table and the real ranks for all 470 questions. The next lesson tests the other way to shrink vectors.
nomic-embed-text v1.5
Its model card calls it resizable and trained with Matryoshka Representation Learning. In the lesson it lost 10 of 441 top-5 answers at 256 of 768 numbers.
bge-m3 (BAAI)
Its model card says nothing about cutting. In the lesson it lost 25 of 439 top-5 answers at 256 of 1,024 numbers, and its first and last numbers did about equally well.
Ollama
The dimensions setting on its embed endpoint keeps the first N numbers and rescales them to length 1, for any model.
OpenAI text-embedding-3
These models accept a dimensions parameter that returns a shortened embedding, the same idea offered through an API.
MRL is a way of training an embedding model. Each example is scored with the full vector and with several shorter front parts of it, and the model is adjusted to do well on all of them. To do well with a short front part, the model learns to put the most useful information first, so the vector can later be cut short.
Pick a model whose card says it was trained to be cut, then measure at the size you need. In the lesson, bge-m3 was better at full size, 356 to 339 ranked first, but nomic-embed-text, which was trained with MRL, was well ahead at 128 numbers, 297 to 254.
You can cut any vector, and Ollama's dimensions setting works on any model. Whether search still works depends on where the model puts its meaning. bge-m3, not advertised as Matryoshka, lost more than nomic-embed-text at every cut in the lesson.
With fewer numbers, texts look more alike, so all scores bunch near the top. In the lesson, the right text scored 0.667 at 768 numbers and 0.802 at 16, but at 16 another text scored 0.847 and won. Judge a cut by which document wins.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
Semantic Search
Search that matches meaning instead of exact words. Text is turned into embeddings, and the results are the documents whose embeddings sit closest to the query's.
Quantization
Storing model weights in fewer bits, such as FP8 or INT8, to shrink memory and speed up inference. Modern methods lose little quality on most tasks.
Cosine Similarity
A measure of how close two embeddings are, based on the angle between them rather than their length. The standard way to rank how related two pieces of text are in vector search.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
This page explains the idea. The "Shorter Embeddings: Keep the First Numbers, Measure What You Lose" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Shorter Embeddings: Keep the First Numbers, Measure What You Lose lessoncourse 2
Take models from notebook to production, with labs on real models.