Answer It Twice
Every eval in this chapter grades the answer. None of them asks what the score would have been with no documents at all.
That is the whole lesson. The rest is how it was measured and what to do about it.
Here is why that question matters. A RAG system has two parts that can produce a good answer. One is the retriever, which finds documents and puts them in front of the model. The other is the model itself, which already holds a lot of knowledge from its training. When the pipeline scores well, you do not know how much of that score each part produced, unless you measure it.
Teams usually spend their effort on the retriever: better , better chunking, a reranker. They judge that work by the pipeline score. But if a large part of the score would be there with no retriever at all, then the score overstates what the retriever is doing, and a change in the model can move the score without any change in retrieval.
The test that answers this is called an ablation: remove one part of the system, run the same eval, and see how much the score drops. Here I remove the documents. On the same 20 questions, the answers with no documents scored 6.7% on my measure, and the answers with the top 5 documents scored 15.1%. So 6.7 ÷ 15.1, about 44% of the reported number, was already there before retrieval ran. I show how that was measured, including a mistake in my first version that I found by reading the answers, and what to report instead.