The Judge and the Candidate Are Usually Relatives
The cheapest judge is the model you already have running. It is deployed, the client library is already imported, and it is good enough to grade with. So the judge ends up being that model, or its bigger sibling, or the same lab's newest release.
And the thing it grades is usually that same model's output, because that is what you are trying to improve.
Think of a cooking contest where one of the judges also cooked one of the dishes. Maybe that judge scores their own dish higher because it is theirs. Maybe their dish really is better. From that one judge's score sheet you cannot tell which. You need a second judge, one with no dish in the contest, tasting the same two plates. This lesson is that second judge, applied to language models.
If a judge favours its own family, every A-versus-B number you report is tilted before the comparison starts. The tilt is invisible, because what comes out is a quality score and a quality score is what you were expecting to read.
So the question is worth an afternoon. The trouble is that the experiment everybody reaches for cannot answer it, however carefully it is run.
In this lesson I run the test on two small open models, qwen3:4b and llama3.2:3b, on a laptop through Ollama. Each one writes answers to the same 50 questions, and each one judges every pair. I explain why one judge cannot answer the question, how two judges can, and how to check the result by hand. Then I go back to the saved data and show two problems in my own run that the headline numbers hid. Those problems are part of the lesson, because they are the kind you will meet in yours.