One Bad Night or Three in a Row: How Alert Rules Trade False Alarms Against the Days It Takes to Notice a Break

Think of a smoke alarm fixed to the kitchen ceiling, a little too close to the toaster. Every morning someone makes toast, a thin wisp of smoke drifts up, and the alarm shrieks. Everybody in the house knows the sound by now. Somebody waves a tea towel at it, it stops, and breakfast carries on.

After a few weeks of this, one of two things happens. Either someone takes the battery out, or everybody simply stops looking up when it sounds. Both are reasonable reactions to an alarm that is nearly always wrong. Both are dangerous on the one morning when the smoke is real.

There are a few ways to make the alarm more bearable without moving it.

You could ask it to sound only when the smoke lasts. A puff from the toaster clears in a minute; a real fire does not. That means waiting before you shout, and on the day of a real fire you lose that minute.

You could ask it, once it has sounded, to keep sounding until the air is properly clear, not just a little better. That stops it from switching on and off and on again as the smoke drifts about, so you hear one long alarm instead of six short ones.

Or you could ask it to stay quiet for a few minutes after it stops, so one burnt slice does not set it off three times.

Each of these choices makes the alarm quieter on ordinary mornings. Each also changes what happens on the morning of a real fire. The question of this lesson is what that trade really costs, measured on a live model rather than a kitchen.

This is lesson 9 of the chapter on watching a model after it goes live.

The data is real and public: the sales of a UK online gift shop from December 2009 to December 2011, published as the UCI Online Retail II dataset. I replay it as if the course's model had gone live in that shop on 1 July 2011, so every date in this lesson is a date in that shop's history, not today's date.

The model is the one from the features chapter's lesson 1. It reads six numbers about each customer and gives each one a score for how likely they are to buy in the next 30 days. Every night at midnight a job scores every customer the shop knows, from 1 July to 9 December 2011.

Lessons 1 to 6 of the chapter each built watchers for that nightly job. Lesson 1 watched the inputs, the scores and the late answers. Lesson 2 wrote rules from what each column means. Lesson 3 compared each night with last week. Lesson 4 tested drift against the training rows. Lesson 5 watched the model's own scores, and lesson 6 watched its ranking with answers of different lengths.

Every one of those watchers answers the same question each night: did tonight cross my line? None of them answered the next question, which is the one a person on call actually lives with: when should that crossing turn into a message that wakes somebody up? Lesson 6 ended on exactly that point. A single ring near a line was weak evidence, and it promised that a later lesson would measure rules that ask for more than one bad night. This is that lesson.

Lessons 7 and 8 of the chapter measure other things (how many answers a drop needs, and slices of customers) and are not used here; this lesson uses the watchers of lessons 1 to 6.