Cutting a Book Into Index Cards
Let me start with a picture.
You have a thick book and a stack of index cards. You copy the book onto the cards so you can find things fast. How much text goes on each card?
Put one sentence on each card and you get thousands of cards. Each card is sharp, but it has lost what comes before and after it. Put a whole chapter on each card and you get a few cards, each one a blur of many topics.
Search systems for AI apps face exactly this choice. Before they can search your documents, they cut them into pieces called chunks. Lesson 02 of this chapter drew search quality against chunk size as a hill. It was bad when chunks were tiny, best in the middle, and bad again when they were huge. It said the numbers were only illustrative, and it measured chunk size in tokens. This lab cuts by words, which are easier to count and check.
This lesson measures the real shape, on this course's own lessons, with the course's own quiz questions as the searches.
The Words You Need First
Please read this slide slowly if any word is new. Every slide after it uses these words.
Chunk. A piece of a document, cut out so it can be searched on its own. Here, chunk size is counted in words.
Overlap. Letting each chunk start a little before the last one ended. Then a sentence cut in two at the edge still appears whole in one chunk.
Retrieval. Finding the chunks that best match a question. It is the "R" in RAG, : find the right text first, then let a model answer from it.
. A list of numbers that stands for the meaning of a piece of text. Texts with close meanings get close lists. An embedding model makes them; this lesson uses nomic-embed-text, a free model that runs on a laptop.
Keyword search. Ranking chunks by the words they share with the question, with no model at all. The version used here is called BM25, a standard recipe that gives more weight to rare words.
Model. A program that has learned a skill from a lot of example data. Here, the skill is turning text into an embedding.
Ollama. A free tool that runs models on your own computer. The lab sends it text, and it sends back embeddings.
Vector. Another name for one embedding, the list of numbers. A search system stores one vector for every chunk.
Cosine ranking. Comparing the question's vector with every chunk's vector. The chunks are ranked by how similar the two lists of numbers are. More similar lists mean closer meanings.
Token. A small piece of text a model reads. In plain English, a word is about 1.3 tokens, so a token is roughly three quarters of a word. Code and formatting use more tokens for each word.
Top 5 and top 1. The top 5 are the five chunks a search ranks highest. A search "hits" when at least one of them comes from the right lesson. The top 1 is stricter: only the single highest chunk counts.
Score. The share of questions that hit. A score of 0.9 means 90% of the questions found the right lesson.
Baseline. The simplest method you could use instead. A new method is only worth its cost if it beats the baseline.
By luck. When two methods are compared question by question, some questions go one way and some the other. The lab asks how often a split this uneven would happen if both methods were equally good. It is like counting heads in coin tosses. This is called a sign test. A result that luck gives only once in hundreds of tries is hard to explain by luck.