The Build Went Red and Nothing Had Shipped
The nightly eval fails. The gate says the pass rate dropped. Somebody checks the model version, the prompt, the golden set and the retrieval config, and every one of them is byte for byte what it was yesterday.
That morning is the subject of this lesson.
The previous lesson moved a number by editing one sentence of the rubric. This one does not edit anything at all.
Why does this happen? An LLM judge produces its verdict by generating text, one token at a time. Unless you tell it otherwise, that generation can involve a random draw at each step. So the same prompt, sent twice, can come back with two different verdicts. Each single verdict is a small coin toss only for the answers the judge is unsure about. But a pass rate adds up many verdicts, so a few coin tosses are enough to move the headline number from one night to the next.
This matters because most teams read the nightly number as a fact about the system. A release gate, the rule that fails the build if the pass rate drops, treats every drop as a signal. If some of the drop is the judge disagreeing with itself, the gate will sometimes fail builds for no reason, and the team will spend mornings looking for a change that does not exist.
In this lesson I measure that disagreement on a judge you can run, show exactly what it does to a release gate, and work out by hand what it costs to reduce it. The short version: at temperature 0 the noise on this setup was zero, and at temperature 1 the same eval moved 6.7 points with nothing changed.