Thirty Cases Is Not a Coverage Number
Every eval set is described the same way. How many cases. Thirty, three hundred, three thousand.
That number says how much work went in. It says nothing about what the set can see.
Join the set against the thing it is testing and a different question appears. Of the documents your retriever can actually return, how many has any case ever pointed at?
On the thirty traces from the last two lessons, against the fifteen documents they search, the answer is ten.
Think of a school exam. A teacher can write a hundred questions, but if all of them come from chapters one to four, the exam says nothing about chapter nine. A student could know chapter nine perfectly or not at all, and the mark would be the same. "A hundred questions" is the size of the exam. "Four chapters of nine" is its coverage. Nobody would accept an exam report that gave only the first number, yet that is how most eval sets are reported.
In this lesson I measure coverage on the same thirty traces as the last two lessons. The method is one join between two lists you already have: the list of documents in the index, and the list of gold documents in the eval set. There is no model, no new labelling, and no new run. The result is a short list of documents that the eval set cannot see, and the lesson is about what that list means and what to do with it.