A judge prompt is built from a record. The record holds the question and the two answers. It also holds everything else about the run: which model wrote which answer, when, under what config, and whatever note the person who set it up left behind.

The template interpolates the record. Nobody reads the rendered string, because the template was tested when it had two answers in it and nothing else.

This lesson adds one line to a judge prompt and measures what happens. The line states no opinion about either answer. It moved 24 of 60 verdicts.

Here is the whole experiment in one picture. Same question, same two answers, same instruction, same judge at temperature zero.

Read the highlighted line again. It contains no judgement. It does not say the answer is better, or newer, or more expensive to produce. It says where the text came from, which is true and which the harness knows.

A leak test is easy to run and easy to run badly. Four choices decide whether the number means anything, and the fourth is the one that separates this from an anecdote.

The third one is worth slowing down on. The line always points at the answer the judge did not choose without it. That makes the null concrete: a judge that ignores the line gives the same verdict twice, so every change is attributable to the line and nothing else.

Before adding anything, run the plain prompt on all 30 tie items in both slot orders and look at what comes back.

That chart is the control, and it is a control because it could have come out anywhere. A judge that always picked the first slot would have put all 30 items in the right-hand bar. A judge that always picked the same wording would have put all 30 in the left.

The 50% is a coincidence of two real behaviours cancelling, not a shape the arithmetic forced. That distinction comes up again at the end of this lesson, for a worse reason.

Now add one line, four different ways, to the same prompt.

The middle row is the finding. A sentence that states no opinion at all, only which of the two answers came from the system being evaluated, changed 24 of 60 verdicts.

The placebo row is what makes that readable. A sentence of the same shape, in the same position, naming neither answer, moved 4. So this is not about the prompt being one line longer. It is about the line naming one of the answers.

The whole experiment is 380 calls to a 4B model on a laptop. The control comes first, the power comes before the result, and the verdict at the end is computed from the numbers above it.

Two things in that output are deliberate and worth copying.

The power table is printed before the result. Read afterwards, a power calculation is an excuse for a null. Read first, it tells you whether the run is worth doing.

And the verdict in section 5 is generated from the numbers, including the branch that says "not shown at this size". A script that prints a fixed sentence about its own result is not reporting, it is asserting.

The first version of this lab tested 24 of 60 against 0.5 and got p = 0.155, which reads as "nothing shown". That number is real and the test is wrong.

The point is worth running rather than reading, because it is the same four counts either way.

Look at the first row of the middle column. Testing against a coin calls "nothing happened" the most significant result in the table. 0 of 60 is exactly as far from half as 60 of 60 is. That is what the wrong null does, stated as loudly as it can be stated.

The second half builds a judge that always answers A, which is as biased as a judge can be, and runs the old broken control against it. It still prints 50.0%.

The null matters more than the test. A deterministic judge given the same prompt twice returns the same verdict, which this run confirmed 60 times out of 60. So a line that does nothing produces zero changes, not thirty.

Against a null of zero, 24 changes out of 24 discordant trials is p far below 0.001. Same data, same arithmetic, opposite conclusion, and the only difference is what "nothing happening" was taken to look like.

Tie items answer whether the judge has a preference. They cannot tell you whether the preference survives contact with a right answer.

The clear pairs can. In those, one answer really is better, and the line points at the other one.

Without the line the judge is competent: it picks the worse answer 3 times in 40. With the line it picks the worse answer 29 times in 40.

The paired view is the clearest one. The line broke 26 trials the judge had got right and fixed none. There is no trial anywhere in the set where the opinion helped.

And it is not 100%, which is the part worth keeping. Correctness held about a quarter of the time, so this is not a model that stopped reading and started echoing. It read both answers, knew which was better, and went with the suggestion anyway.

The failure mode is not noise. It is a bias with a stable sign, and the sign points at whatever you labelled as the thing under test.

That is the part that makes it dangerous rather than merely annoying. Re-run the eval and you get the same number back. Stability reads as reliability, and here it is neither.

If you keep one number from this lesson, keep that one. The opinion condition is the dramatic result and the provenance condition is the one that is probably in your repository right now.

Nobody writes the leak on purpose. It arrives through four normal things.

Every step in that chain is something a reasonable person does. Passing the whole record to the template is convenience. The template interpolating what it was given is correct behaviour. Nobody in the chain decided to tell the judge which answer was the candidate.

What happened here is documented, and the documentation is about much larger models than the one in this lab.

The third note is the uncomfortable one. Part of this comes from the preference data: people rated agreeable answers higher, and the training picked that up. It is not a bug that a patch removes.

For an eval, the safe assumption is that your judge has some of it. The cheap thing to do is measure how much, rather than argue about whether.

The first run of this lab reported a control at exactly 50.0%, which looked beautifully behaved.

The old control counted, per item, how often the no-opinion verdict matched "the hinted letter" across four trials. Two of those four trials used the identical prompt. They differed only in where the hint would have pointed, so exactly one of every pair matched by construction.

It would have printed 50.0% for a judge that always answered A. It measured nothing, and it passed every automated check in this repository, because every check counts things and this one counted correctly.

The test that catches it is one question: what would this control print for a judge that ignores me completely? If the answer is the number you got, you have learned nothing.

The previous lesson made a rule about writing the comparison down before the run. This lesson changed four things about its own method after seeing the first numbers, so it owes you the distinction.

The shuffle is not the point. The point is the third line: your code keeps the mapping and applies it after the verdict comes back, so the judge never sees a name.

The one I deliberately did not test is the defensive instruction, the line telling the judge to disregard any note about provenance. It may work. It is also one more sentence about provenance, in a prompt where such sentences move 40% of verdicts. So I would want it measured before trusting it, and measuring it is this same experiment run once more.

A judge grades what is in front of it, and that includes the parts of the prompt that are not answers.

The fix is three lines and the audit is one minute. What makes it worth a lesson is that nothing in the chain looks like a mistake while you are building it. The number that comes out the other end is stable enough to be believed.