Every eval set is described the same way. How many cases. Thirty, three hundred, three thousand.

That number says how much work went in. It says nothing about what the set can see.

Join the set against the thing it is testing and a different question appears. Of the documents your retriever can actually return, how many has any case ever pointed at?
On the thirty traces from the last two lessons, against the fifteen documents they search, the answer is ten.

Five of the fifteen are never the gold answer to any case: D04, D11, D13, D14 and D15.
That is a third of the index. If one of those documents is stale, badly chunked, or says the opposite of what the current policy says, every eval run passes and nobody hears about it.

It is worse than that, and the second count is the one worth sitting with. Those same five documents never appear in any trace at all, not even as a losing candidate. They have not been ruled in or ruled out. They are simply absent from the evidence.

A document that loses a comparison has been tested. It was a candidate, it scored, and it came second. You know the retriever can see it.
A document that never appears has not been tested at all. You do not know whether it is reachable, whether its chunking is sane, or whether its puts it anywhere near the questions it should answer.
A pass rate cannot tell these apart. Both look like the absence of a problem.


Nobody decided to skip five documents. The shape comes from how the set was built.

Cases come from questions. Questions come from what users asked, or from what the team could think of. Both are drawn from the parts of the product people already use.
Documents that answer rare questions therefore get no cases, and documents added after the set was written get none either, because nobody went back.


The set is not wrong. It is a sample of your traffic, and it inherits traffic's shape, including the long tail traffic does not reach.
The code below joins the traces against the document set. It counts three things. Which documents are a gold answer, which appear anywhere at all, and how the cases are spread across the ones that do get tested.

Section two is the one to sit with. The same five documents appear in both lists, which means they were not beaten by anything. They were never in the room.


Report documents covered, next to cases run. Two numbers instead of one. "30 cases over 10 of 15 documents" is a sentence that tells you where you are; "30 cases" is not.



Treat a new document as a case that does not exist yet. When something is added to the index, it starts uncovered. Making that visible at ingest time is cheaper than discovering it during an incident.

4 questions - Score 80% to pass
Your eval set has 300 cases and your index has 4,000 documents. What does the case count tell you about coverage?
Five documents never appear in any trace, not even as a losing candidate. What do you know about them?
Why does an eval set built from real traffic tend to leave part of the index uncovered?
What is the right target for index coverage?

Both halves of the join are already in your system. The index can list what is in it, and every result already says which document it came from. Nobody is missing a tool here; the query has simply never been written.

Do not chase 100%. A long tail of rarely-read documents is normal, and writing a case for every one of them is not a good use of anybody's week. The useful move is deciding which uncovered documents matter, and being able to see the list at all.

Watch the concentration too. Thirty cases spread over ten documents is a different set from thirty cases on two, and the case count is identical in both. The distribution is in the same query that produced the coverage number.


The third number is the one on every dashboard, and it is the only one of the three that cannot answer the question. Run the join once and you will know within minutes whether your set has this shape too.