LLM-as-Judge and Its Failure Modes

A new version of the support assistant averages 4.6 on the quality rubric. The old one averages 4.1. An LLM read every answer and scored it from 1 to 5, the gap looks real, and the number is clean enough to ship on. Before trusting it, swap the order the two answers were shown to the judge and run the grading again. The gap can shrink, vanish, or even reverse without a single word of either answer changing. The judge has a bias, and the score you were about to ship on was partly measuring position, not quality.

Two weeks later a customer complaint pattern says the opposite. Someone reruns the eval, and this time, on a hunch, they swap which answer the judge sees first. The scores flip. The old version now wins. The judge had not been measuring quality at all. It had been favoring whichever answer sat in the first slot, and the "4.6 versus 4.1" was a coin the team had been reading as a compass.

Nothing errored. No exception, no failed request, no red dashboard. The judge produced a clean number every time, and the number was confidently wrong. That is the danger of LLM-as-judge in one sentence: it hands you a precise score with unknown accuracy, and precision reads as truth. A person who wrote "4.6" by hand would hedge, second-guess, add a note. The model writes 4.6 with the same flat confidence every time. Whether the answer was excellent, mediocre, or the judge was simply guessing. That even tone is exactly what makes the failure so easy to miss.

The judge was never broken. It answered every call and returned a tidy number. The number just did not mean what everyone assumed it meant.

This lesson is the full catalogue of how that happens. The previous chapter on evaluating LLMs introduced the judge as a method and named a handful of biases in passing. Here we go the other way: every documented failure mode, the mechanism that produces it, and the specific fix that removes it. By the end you should be able to look at any judge setup and predict which modes it is exposed to. You test for them, and then decide whether the number is a measurement or a guess.