Let me start with a picture from school. Say a teacher writes an exam by copying sentences straight from the textbook. A student who only matches words can find every answer, without understanding anything. The exam looks easy. Real life is not like that exam.

Now here is one result from a small experiment I ran. I will call the experiment "the lab". After this slide, I will explain every word.

I took 60 documents. For each document, a language model wrote two questions that the document answers. (A language model is the kind of AI that writes text, like the one behind ChatGPT.)

The first question uses the document's own words. It sounds like a test question an engineer writes after reading the page.

The second question asks about the same problem in everyday words. It sounds like a real person who does not know the technical words yet.

Then I asked three search systems to find the right document for each question. A search system is a program that takes a question and returns the documents most likely to answer it.

With questions in the document's own words, all three systems found the right document every single time. 100 percent.

With questions in everyday words, the same systems found it between 35 and 63 percent of the time.

The documents were the same. The search systems were the same. Only the questions were different.

What does that mean for your own tests? If the questions in your test were written by someone who read the documents first, your score is measuring the first kind of question. Your real users ask the second kind.

Before we go further, here are the words this lesson uses. If you already know them, skip ahead. If you do not, please read this slide slowly. Everything after it depends on it.

Document. One page of text in your knowledge base. A knowledge base is the collection of pages your AI app can search, like a company's help pages. Here, a document is one lesson from this site.

Search system. The program that takes a question and returns the documents most likely to hold the answer. Engineers also call it a retriever. Many AI apps are RAG systems. RAG stands for Retrieval-Augmented Generation. A RAG system searches your documents first. Then it gives the best ones to a language model, which writes the answer. Search is the first step. If search misses, the model never sees the right document.

Top 5. The first five documents the search system returns.

Found. Here, a question counts as found when its right document is somewhere in the top 5. The score of a search system is the share of questions it found. Engineers call this score "recall at 5".

Test set. A fixed list of questions, each with its right document written next to it. You run the same list every time you change the system, so you can compare scores. Some teams call it a golden set.

Offline and online. Offline means you test on your test set, before any user sees the change. Online means you measure with real users, after the change goes live.

Here is how most teams build their first test set. It is not a bad process. It is the normal one.

Someone opens a document. It may be an engineer, or it may be a language model you asked to write questions. They read the page, and they write a question that the page answers.

Because they just read the page, they use the page's words. They write "data redundancy" (keeping extra copies of data) because the page says "data redundancy".

A real user has not read the page. That is why they are asking. So they cannot use the page's words. They write "how do I stop losing my files".

So the test set and the real users are not asking the same kind of question. The test set copies the technical words of the answer. The user does not know those words.

Here is a real pair from the lab. Both questions are about the same document, the lesson on data redundancy.

The first question: "What is the primary purpose of data redundancy in the context of hardware failures?"

The second question: "How can I make sure my important files don't get lost if my computer crashes?"

Both questions ask the same thing. But look at the words. The first one says "data redundancy" and "hardware failures", which are the words on the page. The second one says "files", "lost" and "crashes", which a beginner would say.

For the second question, one search system put the right document in 3rd place, so it was found. Another search system put it in 120th place. At 120th place, a user would never see it.

The lab compares three search systems. You need to know what each one does, because they fail in different ways.

Keyword search. It looks for the exact words of your question inside each document. A document that uses the same words scores high, and rare words count more than common ones. The common method is called BM25. It is fast and it is very good when you know the right words.

Meaning search. It turns each document and each question into a list of numbers, called an embedding. Texts with similar meaning get similar numbers, even when they use different words. Then it returns the documents whose numbers are closest to the question's numbers. Engineers call this dense search. Here it uses an embedding model called nomic-embed-text. This is a different kind of model from a language model. It does not write text. It only turns text into numbers.

Both. It runs keyword search and meaning search. Each one gives a list, from best to worst. Then it merges the two lists into one, by position: a document near the top of either list ends up near the top. Engineers call this hybrid search.

You might guess already which one struggles with everyday words. Keyword search needs your words to match the page's words. Everyday questions do not.

Here is exactly what the lab did, step by step.

1. The documents. 818 lessons from this site went into the search index. The search index is the collection the search systems look through. The search can return any of the 818.

2. The pick. The lab picked 60 of those documents at random, using a fixed seed. A seed is a starting number for the random pick, so the same seed always picks the same 60 and anyone can repeat the run. All 60 happened to be lessons from this site's System Design course, not the AI Engineering course.

3. The questions. For each document, a small language model called qwen3:4b wrote two questions. The model saw the title and the first 900 characters of each page. For the first question, it was told to use the page's own technical words.

   For the second, it was told to write like a beginner who does not know the words yet. It was also given a list of the document's ten most special words, and told not to use any of them. A special word is one that appears often on this page but rarely on other pages.

4. The search. Every question, all 120 of them, went to all three search systems.

5. The mark. A question was marked found if its document was in the top 5.

Each document gets two questions. That design matters later, so keep it in mind.

Before looking at any score, the lab checks one thing. Are the everyday questions really written in different words? If the model quietly used the page's words anyway, both kinds would be the same kind, and the whole lab would mean nothing.

So the lab measures word overlap. For each question, it takes the important words (it skips small words like "the" and "how") and counts how many of them also appear in the document.

For questions in the document's words, 68 percent of their important words appear in the document. For everyday questions, 29 percent.

And it holds pair by pair, not just on average. In 53 of the 60 pairs, the everyday question shared fewer words with the document than its twin (the other question about the same document). In 5 pairs it shared more. 2 were tied.

One more honest detail. The model was told not to use ten special words. It still used at least one of them in 9 of the 60 everyday questions. Those 9 were easier to find, for all three systems. Out of every 10 such questions, meaning search found about 8, against about 6 in 10 for the questions that followed the rule. Keyword search found about 8 in 10 against about 3 in 10. Both combined found about 9 in 10 against about 4 in 10. So the gap you will see next is probably a little smaller than it would be if every question had followed the rule.

Some tests use a language model as a judge. A judge is a model you ask to grade another model's answer. Earlier lessons in this chapter showed that a judge can change its grades when you change small things. So this lab uses no judge at all.

Each document has an id, like a file name. The right answer for a question is the id of the document it was written from. That id was chosen before any question was written.

So marking is simple. Is that id in the top 5? Yes or no. No model decides it. So the marking cannot change from one run to the next.

This is the real output of the lab's report. It prints the word-overlap check first, then the scores.

Read it from the top. The first part is the word-overlap check. Its last line says "held", which here means the check passed: the two kinds really are different. The second part is the table of scores you will see on the next slide, printed by the lab itself.

Here are the scores. Each number is how many of the 60 questions had their right document in the top 5.

Read the left column first. If your test set only had questions like these, every system would score 100 percent. You would think search works perfectly.

Now read the right column. Keyword search, the one that needs matching words, drops the most: to 35 percent. That means it missed almost two out of every three everyday questions.

Sixty questions is not a huge number. So we should ask: could a gap this big happen by luck?

Here is a simple way to think about it. Remember, every document has two questions. So we can compare each document with itself.

For a lot of documents, both questions were found. For some, both were missed. Those documents tell us nothing about which kind of question is harder, so we set them aside.

What is left are the documents where the two twins disagreed: one question was found and the other was not. For meaning search there were 22 of these. In all 22, the question in the document's words was found and the everyday one was missed. Not once the other way round.

If the two kinds of question were equally hard, each of those 22 would be like a coin flip. It could go either way. Getting the same side 22 times in a row by luck has a chance of about 1 in 2 million.

For keyword search it was 39 in a row. For both combined, 30 in a row. Those are even less likely by luck.

This way of counting has a name. It is called McNemar's test. Be careful: here "test" means a way of checking for luck. It is not a test set.

McNemar's test is the right check whenever each item gets two yes-or-no results, like each document here: found or not found, for each of its two questions. It only looks at the items where the two results disagree. You will see its result written as a number called a p-value, often shown as "p =" followed by the number. A p-value is the chance of seeing a difference at least this big if there were no real difference. A small p-value means "very unlikely to be luck".

So this is very, very unlikely to be luck. Everyday questions are much harder for all three systems.

Now here is the part that should worry you most.

Say you are choosing between these three search systems. You run your test set. Your test set was written from the documents, so it is full of questions in the document's words.

All three score 100 percent. Your test says they are equally good. In many real systems, keyword search is the simplest and cheapest to run. (This lab did not measure cost.) So you might pick it.

But on everyday questions, keyword search is the worst of the three, by a lot. Your test could not see it, because every question in your test already contained the right words.

When every option scores 100 percent, the test cannot tell them apart. This is called a ceiling. The test is too easy, so everything hits the top, and the differences you care about are hidden above it.

So let us compare the three systems on the everyday questions only. Again we compare them question by question, because all three answered the same 60 questions.

Meaning search against keyword search. 24 questions were found only by meaning search. 7 were found only by keyword search. If the two systems were equally good, a split like this would happen by luck about 3 times in 1,000 (p = 0.0033). So meaning search really is better than keyword search on everyday questions.

Both combined against keyword search. 11 against 2. About 2 in 100 by luck (p = 0.022). Combining really is better than keyword search alone.

Meaning search against both combined. 15 against 7. About 13 in 100 by luck (p = 0.13). That is too likely to be luck. Meaning search looks ahead here, but 60 questions cannot prove it. I will not tell you it wins.

That last line is important. Not proven does not mean the two are equal. It means 60 questions are not enough to decide. You would need more questions.

Before running the lab, I wrote down what I expected. This is good practice: you write your guess first, so you cannot change it after you see the numbers.

I expected two things.

First, that meaning search would beat keyword search on everyday questions. That held.

Second, that keyword search would win on questions in the document's own words, because those questions share so many words with the page. That did not happen. It could not happen, because all three systems scored 100 percent on those questions. There was no room left for anyone to win.

I am showing you this because it is the same ceiling from the last slide, and it caught my own guess too. An easy test does not just hide differences from you. It hides them from everybody.

Real test sets are usually a mix. Some questions come from people reading the documents. Some come from real users. So what score would you report for different mixes?

The lab can answer that, because it has both kinds for the same documents. It just mixes them in different amounts.

Take meaning search. If none of your test questions are in the document's words, it scores 63 percent. If half of them are, it scores 82 percent. If all of them are, it scores 100 percent.

Same system. Same documents. Same search. The score moves from 63 to 100 only because of who wrote the questions.

So when someone shows you a search score, the first question to ask is not "how good is the system". It is "who wrote the questions".

One more honest note. In this lab, the lines cannot cross. Every system scored 100 percent on the questions in the document's words, so the three lines only meet at the right-hand end. Everywhere else, meaning search is highest and keyword search is lowest. (Remember that meaning search against both combined is not proven.) So here, the mix changes the score but not the order. In another system, where the systems score differently on both kinds, the order could flip. This lab cannot show that.

This is the real output of the second half of the lab's report: the system-against-system comparisons, and the score at each mix.

In this recording, dense means meaning search, bm25 means keyword search, hybrid means both combined, exact means questions in the document's words, and plain means everyday questions. The last line is printed by the lab, not added by me.

When an everyday question was missed, what did the search return instead? The lab recorded the top result for every miss.

The page names here are lessons on this site. You do not need to know them. What matters is whether the page returned is close to the right one.

Some misses land on a related page. The question about switching to a backup system was written from the lesson on disaster recovery testing. Meaning search returned the lesson on failover first. That is a related page. A user might still learn something from it.

Other misses are just wrong. The question about real-time data updates was written from the lesson on micro-batching. Meaning search returned the lesson on continuous backup.

For meaning search, 5 of its 22 missed everyday questions returned a document from the same chapter as the right one. (A chapter is a group of lessons on one topic.) The other 17 returned a page from a different chapter. A different chapter can still be a related topic, like the first example here. But it is not the page the user needed.

This lab counts a question as found only if its one right document is in the top 5. A related page does not count. That keeps the marking simple and fixed. But in real systems, more than one page can answer a question, like failover and disaster recovery testing. In your own test set, it is fine to list more than one right document for a question.

This box holds the real results: for each of the 60 documents, whether each search system found each of its two questions in the top 5. Press Run. It counts the scores, runs the luck test on the pairs, and prints the score at each mix.

Then try changing SHARE_DOCUMENT_WORDS at the top. It means the share of your test set written in the document's words. Watch the score change while the search systems stay exactly the same.

Questions written from the documents are not bad. They are just for a different job.

Use questions written from the documents when you want to know if a change broke something. They are cheap to make, and they cover every page. If the score on them drops, something has probably broken, and it is worth a look.

Do not use them when you are choosing between two search systems, or deciding if your app is good enough for real users. They are too easy. Every option passes, so they cannot tell you which one is better.

Use everyday questions, best of all real ones, when you are choosing between options, or when you want to know how real users will do.

Do not trust them alone when you have only a few dozen. As you saw, 60 questions could not decide between meaning search and both combined.

Every lab has limits. Here are the ones that matter for this one.

A model wrote the everyday questions, not real people. They take the place of real users here, but they are not real users. Real users make spelling mistakes, ask two things at once, and leave out context. They may be harder still, or easier in ways I did not test.

One knowledge base. These are lessons from this site. Your documents are different. The size of the gap will be different in your system. The direction is what to take away.

Sixty documents. Enough to show a gap this large clearly. Not enough to settle smaller questions, like meaning search against both combined.

One stray sentence. One everyday question came back with an extra sentence from the model at the end: "(This is a simplified version of the question that avoids the forbidden words.)" It went to the search systems like that. I left it in. Removing a question only because I saw the results would be picking the data to get the answer I want. Without that one question, the everyday scores are 64, 36 and 51 percent instead of 63, 35 and 50. Nothing changes.

The banned words leaked 9 times. As I said earlier, 9 everyday questions still used a banned word, and those 9 were easier to find. So the gap measured here is probably a little smaller than it would be if every question had followed the rule.

One setup of each search system. One embedding model, one keyword setup, and one way of merging. Other choices could change the numbers.

Whole lessons, not small pieces. Each document here is a whole lesson. Many real systems cut documents into small pieces, called chunks, before searching.

1. Label every test question with where it came from. Add one field (one extra column) to each question, like source: "from-document" or source: "real-user". It costs almost nothing. You cannot add it later from memory.

2. Report the score for each source separately. Never only the total. A total of 82 percent could be 100 percent on easy questions and 63 percent on real ones. Those are very different systems to ship.

3. Once you have users, add real questions. Your logs are the records your app keeps of what users typed. Pick some of those questions. Spread them across topics, and include questions where users were unhappy with the answer. Remove anything private first, like names, emails and phone numbers. Then someone must write down the right document (or documents) for each one. This takes real work. It is also the most useful work in this whole list.

4. Use enough questions. This lab used 60, and that was not enough to decide between meaning search and both combined. This next part is advice, not something the lab measured: for choosing between close options, aim for a few hundred real questions.

5. Before you have users, write everyday questions carefully. Ask people who have not read the documents to write questions. Or do what this lab did: ask a model to write questions with the document's special words banned. Then measure the word overlap, like the lab did, to check it really worked.

6. Keep part of the real questions hidden. Tuning means changing settings again and again until the score goes up. If you tune while looking at every question you have, the system gets good at those exact questions, and the score stops telling you the truth. So keep some questions aside, and only check them at the end.