The Fix That Was Sitting Right There
The last lesson ended on an uncomfortable finding.
Our judge scored 100 percent against grader A and 57 percent against grader B, on the same 40 answers. Both of them are careful readers, and neither is a person. Grader A is Mistral Large and grader B is Nova Pro. Each was asked to read all 40 answers by hand. Three answers caused the whole gap. Both graders had called those three partial, and they had agreed completely. The disagreement was never about the answers. It was about which box a half-right answer belongs in when you only have two boxes.
Then we noticed something worse. The two graders were asked for three boxes. The judge was asked for two.
Read those two prompts again. They look almost the same, and it is tempting to say the only difference is one word in the list of allowed answers. That is not true, and the figure's last line says so. The list gained "partial", and the rule that defines the answers was rewritten at the same time. The old prompt had one rule: an answer that refuses, leaves out the cause, or says something the documents do not support is not correct. The new prompt defines three boxes instead, and it now calls a refusal wrong only "when the documents had the answer". So two things changed, and we come back to that below.
The judge was being marked against a scoring guide it had never been shown. That is not subtle. It is closer to marking a multiple-choice paper where one of the options was missing from the student's copy.
The fix looks obvious and it is cheap. Each judge answers 40 short prompts, so a run finishes in well under a minute. Give the judge the same three boxes. Score it against the readers' three-way labels. See what happens.
What came back was not what we expected. The reason is more useful than the fix would have been.