An earlier lesson in this course gave you a catalogue of judge failure modes. Position bias, verbosity bias, self-preference, sycophancy, formatting bias. All of them real, all of them documented.

This lesson takes one entry from that catalogue and checks whether it is happening to a judge you can run on a laptop. The answer turns out to be no, which is more useful than a yes would have been.

Verbosity bias is the claim that judges score padded answers higher than short ones carrying the same content. Length reads as effort, and effort reads as quality.

Everything on the left is true. The effect has been measured and the defences exist for a reason.

What it is not is a fact about your judge. It is a measurement on somebody else's model, with somebody else's prompt, on somebody else's answers.

Read the third note carefully, because the rest of this lesson depends on it. Nothing here disputes the paper. The question is what a reader is entitled to conclude about a different model.

A bias test is easy to run and easy to run badly. Four choices decide whether the number means anything.

The first one is the one to be strict about. If the padding adds a fact, an example, or a genuinely useful caveat, then a judge that prefers it is not biased. It is right.

Read the padding. That is the check, and it is not one a script can do for you. Every sentence restates the answer, hedges it, or says something the reader already knew.

Ten tie items, each judged twice, once with the padded answer first and once second.

Thirty-five per cent. The padded answer did not win, and the judge mildly preferred the short version.

That is the point where it would be tempting to write "verbosity bias did not replicate" and move on. Do not do that yet.

Four lessons ago this chapter was about what a small set can and cannot detect. Apply that to the experiment you just read.

Twenty verdicts. The interval runs from 14% to 56%, and a binomial test against a coin toss gives p = 0.26.

So the tie half of this experiment has not shown that the judge is unbiased. It has failed to show that it is biased, which is a different and much weaker statement.

At twenty verdicts, only a very large bias would reliably show up. A real seventy per cent preference would have been missed more than half the time.

I built this experiment and I have to report that half of it cannot answer the question it was built to answer. That is the same trap this chapter spent a lesson on, and knowing about it did not stop me walking into it.

Sixty judging calls against a 4B model running locally. No API account and no GPU.

The code below is the counting and the power check, without the model calls, so it runs on the page.

Padding a tie tells you whether length breaks a coin toss. What actually costs you is length overturning a verdict that correctness had already settled, so the lab runs that too.

Two verdicts out of forty. A binomial test gives p around 1.5 times ten to the minus nine.

That is a completely different kind of null from the one on the previous slide. It is far enough from a coin toss, at a big enough size, that luck does not explain it.

Same lab, same judge, same run, two null results, and only one of them is evidence. Telling those apart is the skill this lesson is really about.

Twice the verdicts is half the story. The other half is that "does padding overturn a correct answer" is a question about something extreme, and extreme effects are easier to rule out than subtle ones.

While we are here, the same judge was measured for position bias with the same design.

Fifty-one point seven per cent of sixty verdicts. A binomial test gives p = 0.90.

Two documented biases, one small judge, neither one present. The pattern is not that the research is wrong. It is that a research finding is a hypothesis about your system until somebody checks.

The right-hand column is not scepticism about the research. It is the difference between reading a literature review and taking a measurement, and only one of those is about your system.

Believing a catalogue entry without checking is not free, and the cost is not obvious.

Step three is the one that matters. A clause was added to a measuring instrument and nothing checked whether it changed any measurement.

Sixty calls. That is the price of replacing a guess about your own system with a number.

A lesson that reports a null has an obligation to say clearly what it did not test.

The padding used here is bland filler. Most reported verbosity bias involves elaboration that adds structure: bullet points, worked examples, headings. That is a different treatment and this experiment says nothing about it.

And the judge is a 4B model. A larger one may behave differently in either direction. What transfers from this lesson is the method, not the result.

Every entry in the catalogue becomes the same question: can I build a test for this, and if I run it, will the result be readable?

Those four steps are not specific to verbosity. They measured position bias in this course with the same shape, and they will measure formatting bias or self-preference with the same shape.

The middle one is worth stealing whatever you are measuring. Asked politely for a single character, this model replies with a sentence. Reading the first bare letter out of that sentence records a verdict the judge never gave.

The tie pairs are the work. Everything else is a loop.

Two documented biases. One small judge. Neither one present.

That is not an argument against the research, and it is not a reason to skip the defences. It is a reason to spend sixty calls finding out which defences you need, before you spend a sprint building them.