Somebody Improved the Rubric
A rubric is the sentence in a judge prompt that says what to look for. This one was vague, so somebody made it clearer. The pull request reads better than what it replaced and the review approved it in a minute.
Next week the pass rate has moved and the team goes looking for a bug in the model.
This story is common because of how an LLM judge works. A judge is a language model that reads an answer and returns a verdict, such as PASS or FAIL. It has no fixed idea of what "good" means. It builds that idea, each time, from the words in the prompt. So when you change the words, you can change the verdict, even when the answer being graded is exactly the same.
That is a problem for anybody who reads an eval number as a fact about the system. If the number can move because a sentence in the prompt moved, then part of the number is a fact about the sentence. You need to know how large that part is, and which uses of the number it spoils.
An earlier lesson in this chapter shows a rubric-sensitivity chart and marks it, correctly, as illustrative: five phrasings, a twenty-point band, numbers chosen to make the shape visible. This lesson measures the real thing, and asks a second question that chart could not.
The size of the band is the part an illustration can carry. What it cannot carry is whether six phrasings disagree only about the level, or also about which answers are better. That second question decides what you can still compare after a reword, and it is the one this lesson was built to answer.
Here is the plan. First I define the words. Then I show the six sentences I swapped and the control that makes the test readable. Then the pass rates, and why the headline spread hides most of the effect, worked by hand. Then the level-or-order question, with the set arithmetic done by hand and in code. After that: what breaks, what survives, the mistakes I see most often, and when you should and should not spend the extra calls.