Name: System Design Masterclass
Availability: InStock

You have read your traces and labelled them. Those labels are now the ground truth for everything downstream: your metrics, your dashboards, the validation of any automated judge you build.

And nobody has checked whether they are any good.

An illustration of two colleagues sitting at opposite ends of one long desk, each reading their own identical stack of papers with a low divider between them, not looking at each other and not talking. Both have a pen in hand

This is the step teams skip, and skipping it is how a whole evaluation practice ends up resting on one person's opinion on one afternoon. The check is simple. Have a second grader label the same traces, without letting them see the first grader's answers, and measure how often the two agree.

A rubric is the scoring guide you hand a grader: the short written rule that says what counts as correct, and what counts as a failure. If two careful graders read the same rubric and still reach different answers, the rubric is not doing its job. It looks like a rule. It is really just an opinion.

Here is the result from the lab, and the interesting part is that we ran it twice.

We graded the same 40 traces twice, with two models from two different companies, under two rubrics.

Both rubrics gave exactly the same raw agreement: 0.93. The graders matched on 37 of 40 traces either way.

If raw agreement were the number to report, you would say the two rubrics were equally good. Kappa says one is much better than the other: 0.69 for the two-choice rubric, 0.82 for the three-choice one.

Here are both runs.

A real terminal screenshot headed the same forty traces with no middle box, titled Raw agreement 0.93 again, chance 0.76, kappa 0.69. A two by two table shows A equals correct with 33 under B correct and 3 under B wrong, and A equals wrong with 0 and 4. Raw agreement reads 0.93, expected by chance 0.76, and Cohen's kappa 0.69, described as substantial agreement on the usual reading

This is the run with only correct or wrong to choose from. The graders match on 37 of 40, but two graders matching by luck would already reach 0.76 here, so kappa is only 0.69.

A real terminal screenshot headed correct partial wrong, titled Raw agreement 0.93, chance 0.58, kappa 0.82. A three by three table shows A equals correct with 28, 2 and 0; A equals partial with 1, 5 and 0; and A equals wrong with 0, 0 and 4. Raw agreement reads 0.93, expected by chance 0.58, and Cohen's kappa 0.82, described as almost perfect agreement on the usual reading

This is the run with a middle box for partial answers. Raw agreement is the same 0.93, but luck alone would only reach 0.58 here, so kappa rises to 0.82.

Count the coloured cells. Thirty-seven in both grids. The grids are not the same shape, and that shape is the whole difference between the two kappas.

Nothing got worse between those two runs. The binary rubric was hiding the disagreement, because it did not offer anywhere to put it.

Inter-Annotator Agreement: Before You Trust Your Own Labels

The rest of this lesson

Continue this lesson

Your Labels Are Data. Nobody Checks Their Quality.