Input Checks: Which Cheap Rules Catch a Broken Table?

Every morning a delivery truck stops at the back door of a small supermarket. The driver wheels in crates of milk, apples and bread. A store worker stands at the door with a clipboard. Her job is to check each delivery before it goes onto the shelves.

She does not open every carton. She has a few quick checks. Some she learned from experience: "we usually get between 40 and 60 crates", "milk cartons are always cold". Others come from what things are, and need no experience at all: "a crate cannot hold minus three cartons", "nothing arrives with a date before it was made".

The leaking crate is easy. Her checks catch it in a second. But one morning the crates are full of the right number of the right-looking cartons, all cold, and every one of them is skimmed milk instead of full cream. None of her checks can see that.

Her checks are cheap, fast and useful. They also have holes. This lesson measures both: which cheap checks catch which mistakes, and how often they raise an alarm when nothing is wrong.

This is lesson 2 of the chapter on watching a model after it goes live. In lesson 1 I replayed the course's model night by night, from its launch on 1 July 2011 to the last day of the data, and broke it four ways. Four watchers, checks that run by themselves every night, looked for trouble. The input watcher, which checks the numbers going into the model, was the fastest for three of the four breaks. But it also rang on every single clean night, because two of the model's numbers count days and simply grew past the training range.

That leaves an obvious question, and it is the question of this lesson. Input checks are the cheapest watcher a team can build. Which kinds of input check are worth having? Which mistakes does each kind catch, and what does it cost in false alarms?

The model is the same one: it reads six numbers about a customer of a real online shop and gives a score from 0 to 1 for how likely they are to buy again in the next 30 days. The nights are the same 162 nights, built the same way. My lab checks that they are equal to lesson 1's stored nights, number for number.

I also build on drift alarms against real harm, which tested shift tests on house prices. Shift tests line up all the values in a column and compare how many are small, middling and large with the same count from the past. This lesson is about something simpler: rules that each single row should obey.