Thirty Traces, Thirty Passes, Nothing Learned
Take thirty real traces. They are the ones captured for the chapter: ten questions, three runs each, with the real similarity score of every document the retriever brought back.
Put a reasonable metric on them. "Did retrieval put the right document in the context window?" It is the first thing most teams measure, and it is the right question to ask.
The answer is yes. Thirty times out of thirty.
That number has a property worth naming. Its variance is zero. The metric never once took a different value, so nothing about it can move, so nothing about it can tell you anything. If the retriever got worse tomorrow, this metric would have to drop below 30 of 30 to say so. But the set contains no case where it ever did, so nobody knows whether it can.
Think of a thermometer that is stuck. It shows 20 degrees in summer and 20 degrees in winter. On a warm day it is "right", but you learn nothing from it, because it would show the same number on any day. A metric with the same value on every case in your set is that stuck thermometer. It may be correct today, and it still cannot warn you about tomorrow.
Most lessons about evaluation ask "how good is my system?". This lesson asks an earlier question: "how good is my test set at noticing a change?". I measure three things on the thirty real traces, and none of them needs a new run: whether each metric can move at all, how close each case was to going the other way, and how big a change in the scores would be enough to flip the answers.