Every RAG eval in this chapter grades the answer. None of them asks what the score would have been with no documents at all.

That is the whole lesson. The rest is how it was measured and what to do about it.

Same 818 documents and same 20 known-item questions as the previous lesson. Same model, same temperature, same seed. The only difference between the three conditions is how many retrieved documents sit above the question in the prompt, and in the first condition that number is zero.

Grading free text usually needs a judge model, and this chapter has spent three lessons on how much a judge moves when you nudge it. Worse, a judge cannot grade the no-document condition on the same scale as the others unless you hand it the same rubric and hope.

So the grade here is a word-list intersection. Take the content words of the gold document, count how many appear in the answer, divide. The list is built from the document alone and never from an answer, so it cannot be tuned to what any answer happens to say.

Its one virtue is the one that matters: an answer written with no documents can be scored on exactly the same scale.

Did the documents change the answer at all? If a system's answers are identical with and without retrieval, nothing else here is interesting.

They changed it a lot. Mean cosine 0.836, only 13% of content words in common, and not one of the 20 answers came back near identical.

That matters for how the rest reads. This is not a pipeline that ignores its context, which is why a rising score looks so obviously like retrieval doing its job.

The first pass gave 10.0% with no documents against 18.2% with the top five, which is a baseline worth 55% of the reported score.

Then I read the answers, and found a confound. This model opens every answer by restating the question. So some of the gold document's words arrive because they were in the question, not because the model knew them or retrieved them. Only 4.1% of the gold vocabulary is in the question, but at these scores that is not nothing.

Dropping every gold word the question already contains: 6.7% against 15.1%. The baseline's share falls from 55% to 44%, and 44% is the number this lesson quotes everywhere else.

The correction is worth more than the number it changed. A confound found by reading the output is the normal way these labs go wrong. The fix was to define the word list more strictly and rerun the arithmetic on answers that were already on disk.

Split the questions by whether retrieval actually found the right document. Where it did, the score rose 14.5 points. Where it did not, 1.1.

That gap is the positive control. A word-list measure that moved the same amount either way would not be detecting retrieval, and a baseline computed with it would mean nothing. This one separates the two groups cleanly, so a small number elsewhere is a finding rather than a blind instrument.

Read that as the case for the retriever, not against it. Where it worked it nearly tripled the score. That shape is exactly what a working component should produce, and it is invisible in the single averaged number.

Section 2 is the control. Section 3 is the question a RAG dashboard never asks. It is also where the headline went from 55% down to 44%, once the confound was removed.

Coverage rose on 15 of the 20 questions. An exact sign test on that gives p = 0.0414, so the lift is real.

It is also modest, and five questions did not improve at all. An average reports neither of those things, which is why the count belongs beside it.

A pipeline score is a sum. It goes up when the retriever improves. It also goes up when the model gets better at answering from what it already holds, and one number cannot tell you which happened.

Step two is the sentence that does the damage, and nobody ever says it out loud. The eval measured a pipeline. The sentence credits a component.

Part 2 is the check worth copying. If the split does not reconstruct the whole, the two halves are not the same measurement and comparing them says nothing.

Part 4 is the one that will catch people out. Every model release raises the no-retrieval floor. A retriever that was earning its cost last year may not be this year. A dashboard showing only the level reports that as no change at all.

Twenty questions and one small model is not a study. The published version is 10 models, 4 augmentation methods and 14,000 questions. It reaches the same shape: retrieval pays where the knowledge is rare, and the unassisted model stays competitive where it is not.

Their conclusion is the operational one. If you know which half of your traffic is which, you can retrieve only where it helps and stop paying for it where it does not. You cannot know that without measuring the floor.

Both answers open by restating the question, which is this model's habit and is part of why the two are as similar as they are. It is also where the confound in the first version of the score came from.

Read past the preamble and the difference is visible: the second answer names a document and quotes a number out of it. That is retrieval reaching the output, and it is what the coverage measure is counting.

Two caveats, both real.

Coverage is not answer quality. A fluent wrong answer that reuses the right vocabulary would score well here. It was chosen because it is the only measure here that a system with no retrieval in it can be scored on at all. That is the whole point.

And 20 questions establishes the direction, not the heights. The sign test carries the claim that the lift is real; nothing here carries the claim that it is exactly 8.5 points.

What survives both is the procedure, and the procedure is the deliverable.