Name: System Design Masterclass
Availability: InStock

Let me start with a picture.

A teacher wants to know if the class is ready for the final exam. So she writes a practice test. But she writes it with the real exam paper open beside her, and copies some of its questions.

The class does well on the practice test. Is the class ready? We cannot tell. The practice test was not a fair preview of exam day, because the exam leaked into it.

A flat illustration of a classroom. A teacher at the front desk writes a test sheet, with an open booklet of questions lying beside it. Behind her, a student reads a test sheet. A calendar on the wall has one day circled. Below the scene: A practice test copied from the exam says nothing about exam day.

Machine learning has the same trap. It is called leakage. The data you test on is supposed to stand in for new data the model has never seen. If anything learned from the test data slips into training, the test score stops meaning anything.

Everyone knows the rule "split first". What is less known is that the leak often hides in a small preparation step that runs before the split, and looks harmless. This lesson measures four of those steps. Some of them lie badly. One barely matters.

Please read this slide slowly if any word is new. Every slide after it uses these words.

A hand-drawn glossary: split, the data cut into rows to learn from and rows to test on; cross-validation, splitting five ways and averaging the five test scores; fold, one of those five parts; column, one measured thing about each row; preparation step, anything done to the data before the model learns; leakage, when the test rows help shape the model; later rows, rows put away until the very end.

Row and column. A dataset is a table. Each row is one example, like one patient. Each column is one thing measured about it, like tumour size.

Label. The answer we want the model to learn, like "malignant" (cancerous) or "not".

Split. Cutting the rows into a part the model learns from and a part it is tested on.

Cross-validation. Splitting the rows into parts called folds, often five. Each fold takes a turn as the test part while the model learns from the others, which I call the learning rows. Those test parts are the test rows. The test scores are averaged. That average is an estimate: a guess, made before launch, of how well the model will do on new data.

Preparation step. Anything done to the data before the model learns: choosing which columns to keep, rescaling numbers, filling gaps, copying rare rows.

Leakage. When the rows used to test a model help shape it. The test score then promises more than new data will give.

Later rows. Rows put away at the very start and not touched until the end. Other books call them the test set or the holdout set. The rest of the rows, used for everything else, I call the building rows.

Seed. A number that fixes one random cut of the data, so a run can be repeated exactly.

Leakage Before the Split: How Pure Noise Scored 93% Accuracy

The rest of this lesson

Continue this lesson

A Practice Test Made From the Real Exam

The Words You Need First