The nightly eval fails. The gate says the pass rate dropped. Somebody checks the model version, the prompt, the golden set and the retrieval config, and every one of them is byte for byte what it was yesterday.

That morning is the subject of this lesson.

The previous lesson moved a number by editing one sentence of the rubric. This one does not edit anything at all.

Every measurement in this lesson comes from running one eval, unchanged, more than once.

The answers are the same 60 the previous two lessons used: the better and the worse answer of 30 items. The rubric is one of the six phrasings from the rubric lesson, "Does this answer actually answer the question that was asked?", which scored 90.0% there. It was picked because it sits far enough from both ends to move either way.

Before reading any movement as sampling, the harness has to be ruled out.

Every repeat got a different seed. At temperature 0 the decoder takes the most likely token every time and the seed has nothing to do, so six different seeds must give six identical runs. They did.

That matters because there is another explanation for everything below, and it is a loose harness. A file read in a different order. A dictionary iterated differently. A race in the client. This rules all of them out without any argument.

Temperature is the setting that decides whether the decoder takes the most likely next token or samples from the distribution. At 0 it takes the most likely one. Above 0 it samples, and two identical requests can come back different.

It is easy to leave unset, and unset does not mean 0. Section 6 of the lab sends one borderline answer twenty times with no temperature and no seed in the request. That is what a judge written from a quickstart looks like. It came back 2 PASS and 18 FAIL. The same answer at an explicit temperature 0 came back FAIL twenty times out of twenty.

So the default on this server is not greedy. If your judge call does not name a temperature, it is running one of the lower rows in that chart and not the flat one.

6.7 points, on a system that did not change.

That is the whole finding in one line. The width is a property of a number in the request body, and at temperature 0 it is zero.

A pass rate counts answers on one side of a line. An answer far from the line contributes the same digit every run. An answer sitting on the line contributes a coin flip.

So the headline is a sum of 48 constants and a dozen coin flips. That is why it moves, and it is also why it does not move very much: the coin flips are a small share of the total.

One answer failed in all six runs and 47 passed in all six. The other twelve are spread across the middle, and every one of them is an answer the judge could not settle with itself.

Section 2 is the control. Section 3 is the one that changes what you do on Monday. Section 6 is the one that decides which row of this chapter you are already standing in.

Two unrelated ways of nudging an eval. One changes the sentence in the prompt and holds the sampling still. The other holds the sentence still and lets the sampling move. They have no mechanism in common.

Seven of the eight answers the rubric rewording could not settle are also answers the sampling could not settle. Under independence you would expect 1.6.

One honest caveat, and it is in the lab output too. The single rubric used here is one of the six the previous lab varied, so the two sets are not independent samples. Read that tail as a distance from chance rather than as a test of a hypothesis anybody posed.

The reading that survives the caveat is the useful one. Some answers are near the judge's decision boundary, and near the boundary any small change in the prompt or in the decoding moves the verdict. Those answers are where your measurement lives, and they are a small fraction of the file.

Take any two of the six runs. Call the first one last week and the second one this week. Nothing happened in between.

There are 30 ordered pairs of six runs. Every drop in any of them is a false alarm, because both sides graded the same answers with the same prompt.

At temperature 1 a gate that fires on a 5 point drop rings on 4 of those 30 for no reason. One that fires on 3 points rings on 8. At temperature 0 no threshold in the table ever fires.

A threshold is not a statement about how much regression you are willing to tolerate. It is a statement about what your eval can distinguish. Below the noise floor it distinguishes nothing, and the only thing it produces is alerts people learn to ignore.

Which is the worst outcome available. A gate that fires often and means nothing is worse than no gate, because it trains the team to click through the one night it was right.

Raising the threshold stops the false alarms and it also stops catching small regressions. That trade is arithmetic, and it is worth doing on your own numbers rather than guessing.

Part 2 is the half people leave out. A team that has been burned by false alarms raises the threshold until the noise stops. Now the gate sleeps through the regression it was installed to catch. Both columns belong in the same table.

Part 3 is the fallback when you cannot pin the sampling. Averaging several runs shrinks the spread roughly as the square root of the number of runs, so cutting the noise in half costs four times the calls. It is real and it is not cheap.

Majority voting is the standard fix and it works. It is measured here over every pair of single runs, and over every way of splitting the six repeats into two panels of three that share no run. Neither side is one lucky pair.

The order to try things in is set by what they cost. Pin the temperature first, because it is free. Repeat only if you cannot.

The control in this lesson is real, and it is the easy case. One machine, one request at a time, weights that do not move.

Two things in that abstract matter here. The models were configured to be deterministic and still moved, so pinning the temperature is necessary and it is not enough on its own. And the cause they point at is batching. That is a property of how the service is run, not a bug somebody is going to fix for you.

Which is the point of the whole lesson. The number to put in your config did not come from here. It comes from running your eval several times on one build and reading the spread.

Two things move a pass rate that have nothing to do with the system being graded. The judge's sampling is one and it is cheap to remove. Which 60 answers happen to be in the file is the other, and it is not.

The eval-power lesson measured the second one. This lesson measures the first. A release gate has to clear both before it can see anything real. Which is why a gate threshold copied from a blog post is a number about somebody else's file.

The order in that flowchart is the order of cost. Re-running the eval is minutes. Reading a git log is seconds. Looking for a regression is a morning, and it should be the last thing you do, not the first.