You Cannot Read Every Answer, So Something Else Has To
The last chapter had you reading traces by hand.
A trace is one saved request. It holds four things: the question the user asked, the documents the system found, the answer it wrote, and a note saying whether that answer was good.
Three of the four come out of the system under test. The fourth is the one somebody has to sit down and produce, and it is the expensive one.
Reading forty of them took an afternoon. It was worth it, and you cannot do it every day.
That afternoon is the most valuable thing in this chapter. It is also the thing you cannot repeat.
So teams build a judge. A judge is a second model whose only job is to look at an answer and say whether it is good. It is cheap and it runs on everything, which is the point.
The disc and the columns are drawn to the same scale. That gap is the entire argument for building a judge, and it is also why the judge has to be cheap.
That is why a judge exists at all. It comes with a trade, and the trade is the reason this lesson exists.
The labels you trust are expensive. That is why you only have forty of them. The judge you can afford to run on everything is small and cheap. And a small cheap model is exactly the kind that gets things wrong.
The careful labels have one job left. Not to grade your system. To grade your judge.
This lesson is about what comes back when you do that, and how to read it. Everything in it was measured. Three judges, the same forty traces, run through scripts/labs/judge-validation, and you can run it yourself.
One function, three arguments that matter. Everything measured in this chapter came out of repeated calls to it.