Imagine a 40-page report. Someone asks you one small question about it. You can hand them the whole report, or you can hand them the one card from a card box that holds exactly that answer.

Search systems face the same choice. The last lesson showed two problems with long text: a model may cut it off, and even when it reads everything, one sentence counts for very little in one long embedding.

The usual answer is to cut each document into small pieces before embedding, one card per idea. In this lesson I measure whether that really helps, on this course's own lessons.

If a word below is new, read its line. Embedding, cosine similarity and search come from earlier lessons in this chapter.

Chunk (or piece). A short part of a document, embedded and searched on its own.

Chunking. Cutting documents into chunks before embedding them.

Chunk size. How long each piece is. Here, 200 words.

Best piece. When a document is split, its score for a question is the score of its closest piece.

Ranked first. The right lesson was the top result.

An index is the stored list of embeddings that a search compares a question with. There are two ways to build one from the same lessons.

Whole: one embedding per lesson. A question is compared with 112 embeddings, one per lesson.

Pieces: each lesson is cut into 200-word pieces, and each piece gets its own embedding. The 112 lessons became 1,823 pieces. A question is compared with all 1,823, and each lesson is scored by its best piece.

The pieces cost more to store: 1,823 lists of numbers instead of 112. The question is whether they find more.

Here is the idea on one short document about three things: response format, rate limits and billing. This is the real file, open in my VS Code.

And this is what it printed for the question "How many requests per minute can I send?"

The whole document scored 0.654. Piece 2, the one about rate limits, scored 0.697. The other pieces, mostly about the response format and billing, scored lower. The piece is about the answer, so it matches better than the whole.

Look closely at piece 2: it starts with "what went wrong." That is the end of the sentence before. Cutting at a fixed number of words splits sentences in the middle. Try SIZE = 10 and SIZE = 80 and watch what changes.

To run it: install Ollama from ollama.com, open it and leave it running. Run ollama pull bge-m3 in a terminal (about 1.2 GB), save the file below as split_and_search.py, and run python3 split_and_search.py (on Windows: python split_and_search.py). You need only Python 3; the "(venv)" in my screenshot is just my own setup, so it says python. If you see "Connection refused", open the Ollama app first.

• Documents: the 112 lessons of this AI course, outside this chapter, with their quizzes removed.
• Questions: the 470 quiz questions inside those lessons, asked exactly as written. The right answer is the lesson each question came from. Because quizzes were removed from the searched text, a question can never find itself.
• Model: bge-m3, with its reading limit raised to 8,192 tokens (the Ollama settings num_ctx and num_batch, from the last lesson) so it can read long lessons. Half the lessons are shorter than 3,295 words and half are longer.
• Two indexes: one of whole lessons, and one of 200-word pieces.

Here are the results, from the terminal.

With whole lessons, the right lesson came first for 286 of 470 questions, and was in the top 5 for 401.

With 200-word pieces, it came first for 356, and was in the top 5 for 439.

That is 70 more questions answered by the first result, from the same text, the same model and the same questions. The only change is how the text was cut.

You might think pieces only help because long lessons get cut off. The test says otherwise. I split the questions by the length of the lesson they belong to.

• Shorter lessons, under 3,295 words: top 5 went from 194 of 240 to 217.
• Longer lessons: from 207 of 230 to 222.

Pieces helped both. And nothing was cut off: with bge-m3 reading up to 8,192 tokens, every lesson fit, the longest at 7,241 tokens (about 1.4 tokens per word). So the whole gain comes from how the text was embedded, not from long text being cut. A quiz question is about one idea, and a 200-word piece is about one idea. A whole lesson is about many ideas at once, and its one embedding stands for all of them.

Looked at question by question, ranked first:

• 255 questions were ranked first both ways.
• 101 were ranked first only with pieces.
• 31 were ranked first only with whole lessons.
• The other 83 were not ranked first either way.

So pieces are better overall, but not always. Some questions are about what a whole lesson is about, and the whole-lesson embedding matches that better than any single piece.

With pieces, 31 of the 470 questions still did not have their lesson in the top 5. They were not spread evenly across the course.

The Evals chapter has 36 lessons, a third of the course, and 166 of the questions. Pieces missed 19 of them, 11%. Every other chapter missed between 2% and 7%.

The Evals lessons are all about one subject, measuring AI systems, from many close angles. My guess is that a question about one of them often finds a very similar lesson first. But this test did not record which lesson came first on a miss, so that is a guess, not a measurement.

This is true in both cases: when many documents are about nearly the same thing, telling them apart is harder, whatever the chunk size.

Pieces are not free. Each embedding from bge-m3 is 1,024 numbers, and each number takes 4 bytes to store.

• Whole lessons: 112 embeddings, about 0.46 MB.
• 200-word pieces: 1,823 embeddings, about 7.5 MB, 16 times as much.

Each search also compares the question with 16 times as many embeddings. For this course that is still tiny. For millions of documents, it becomes a real cost in storage and search time, and later lessons in the course cover how large systems handle it.

You also need to keep the text of each piece, so that you can show it or pass it to a language model.

This lesson used one size, 200 words, and did not test others. Size is a real trade-off:

• Too small: a piece loses the words around it. "It returns 429" (429 is an error code) means little without the sentence before it.
• Too big: a piece covers several ideas, and we are back to the whole-document problem.
• Cut in the middle: fixed word counts split sentences, as in the small example.

The retrieval chapter of this course, Retrieval and RAG in Production (RAG means search plus a language model), measures eight sizes on this site's lessons, with a different embedding model. Its lesson Chunk Size, Measured found that embedding search did best with small pieces and got steadily worse as the pieces grew. It also found that a plain keyword search, with no model at all, scored higher at most sizes.

This box holds the real ranks from the test: for each of the 470 questions, where the right lesson came, whole and with pieces. Press Run.

Then change TOP = 1 to TOP = 5, or TOP = 3.

• Split when documents are longer than a few paragraphs, or longer than the model reads.
• Split when people ask about one detail at a time, which is most questions.
• Keep whole when documents are short, like a product title with one line of description, as in lesson 4.
• Test both on your own questions, as this lesson did. The best choice depends on your text.

Measured: one model, one piece size of 200 words, 112 lessons and 470 of their own quiz questions.

Not measured: other piece sizes, pieces that overlap or follow sentence breaks, other models, or questions written by real users. The quiz questions were written by the same author as the lessons, often in the same words. That probably makes this search easier than it would be with real users' questions, but this test did not measure how much.

1. Run the small example and try different sizes.
2. Split long documents into pieces before embedding.
3. Score each document by its best piece, or return the piece itself.
4. Measure on your own questions, whole against pieces.