Wrong Labels: How Many Can a Model Survive, and Can You Find Them?

Picture a teacher marking a tall pile of worksheets late at night. Most of the marks are right. A few are wrong: a 3 marked as an 8, a correct answer crossed out. By the bottom of the pile, tired eyes make more of these slips.

Now imagine a student who learns only from those marked sheets. Every wrong mark teaches them something false.

A machine learning model is that student. It learns from examples that people have labelled, and some of those labels are always wrong. In this lesson I measure how many wrong labels a model can survive. I also measure whether it matters which labels are wrong, and whether you can find them without re-checking every row by hand.

This is the eleventh lesson of the chapter on data engineering for machine learning. It builds on two lessons, and I link to them rather than repeat them.

Lesson 7, on data labelling, is about how labels get made and what they cost. It explains why even trained people disagree, how to count how often two labellers agree, and how to use their disagreements to improve the written labelling guide. Its lab measures how far a small labelling budget goes. The evals lesson on inter-annotator agreement goes one step further. It shows why a raw agreement count can flatter you, and how Cohen's kappa removes the share of agreement that luck alone would give.

This lesson asks the next question. Once some labels are wrong, what do they do to a model, and can a model help you find them?

What this rebuild corrected. I checked every claim in the earlier version against its source. The full record is in results/ln-factcheck.json.

• It said a random forest lets "300 small decision trees" vote. scikit-learn grows each tree in full, and it averages their probabilities rather than counting votes.
• It called flagging every disagreement "the first step" of confident learning. The confident learning paper calls it the simplest variant and uses it as a baseline to compare against, not as a first step.
• It said a model asked about rows it trained on would vouch for them, so its flags "point almost nowhere". I measured it for this rebuild. That held for the forest and failed for logistic regression.
• It said the lab takes about four minutes. I had no record of that, so I cut it.