Two Documented Biases, Neither of Them in My Judge

An earlier lesson in this course gave you a catalogue of judge failure modes. Position bias, verbosity bias, self-preference, sycophancy, formatting bias. All of them real, all of them documented.

This lesson takes one entry from that catalogue and checks whether it is happening to a judge you can run on a laptop. The answer turns out to be no, which is more useful than a yes would have been.

A judge, in this chapter, is a language model that reads two answers to the same question and says which one is better. Teams use a judge because reading thousands of answers by hand is slow. A judge is fast and cheap, but it is still a model, and a model can have habits that have nothing to do with quality. Those habits are called biases: a steady push toward one kind of answer, whatever the answer actually says.

The catalogue tells you which biases other people found in other judges. It cannot tell you which ones your judge has. So in this lesson I take one entry, verbosity bias, and I test it on qwen3:4b, a small open model that runs on a laptop through Ollama. I also show what the same judge did in the earlier position-bias lab. By the end you will know how to build the test, how to check its numbers by hand, and, most of all, how to tell a null result you can quote from one you cannot.