The Same Judge Got Two Very Different Scores
In the last lesson we checked a cheap judge against careful labels and it looked good. It caught every real failure in the set.
Two reminders before the numbers. A trace is one saved request: the question, the documents, the answer, and a note on whether the answer was good.
And a word about grader A and grader B, because it changes how much you should trust all of this. They are not people. They are two large models, Mistral Large and Nova Pro, run with a careful scoring guide. We treat their labels as the truth here for two reasons. They are far more careful than the small judges we are testing. And two big models cost a few cents, while two experts cost a day.
Both graders see one thing the judges never see: the document that actually answers the question. That is why they are the reference and not the other way round.
In your own work those labels should come from people who know the domain. Everything in this lesson still applies, and it applies harder, because two people disagree in ways two models do not.
The picture is the method. Two readers who can see each other's work are one reader.
Then we checked the same judge against a second set of careful labels.
Each grader gets all 40 answers, plus the document that really answers each question, and sends back its own 40 verdicts. Only then are the two lists compared, answer by answer.
There is no arrow between the two graders, and that absence is the design.
This is the first check, against grader A. The judge caught all 4 real failures and left 32 of the 36 good answers alone, which comes out as 90 percent accuracy.
Read the two together.
Nothing about the judge changed. Same model. Same prompt. Same forty traces. Same answers.
Two report cards for one run. The verdicts underneath them are byte for byte identical.
The share of real failures it caught went from 100 percent to 57 percent.
If you had run only the first check, you would have shipped this judge and told your team it catches everything. You would have been reporting a real number, honestly measured, and you would have been wrong about the thing that matters.
This lesson is about why that happened, because the cause is not what most people guess, and the fix is cheap.