Watching a Model's Own Scores: What They Tell You Before the Real Answers Arrive

Think of a weather forecaster on the local radio. Every evening she says something like "a 20 in 100 chance of rain tomorrow". Nobody can check her that evening. The rain comes, or it does not, the next day.

Still, you can learn a lot from her forecasts alone, before any rain falls. If one evening she says "90 in 100" for every town in the country, in the middle of a dry summer, you do not need to wait for tomorrow. Something is wrong with her, her instruments or the sheet she is reading from.

But some problems she cannot show you. Say the rainy season comes early this year. Her charts do not know it yet, so she keeps saying 20 in 100, and it rains on half the days. Her forecasts look calm and normal. Only the rain gauge, checked day after day, tells you she has fallen behind.

And there is a third kind of change. Suppose she goes on a course and comes back more careful. Now she says 40 where she used to say 20, and 70 where she used to say 50. She still puts the days in the same order, wettest to driest. But the farmer who brings in his hay whenever she says 50 or more now does it three times as often.

This lesson is about the same three things, for a model instead of a forecaster.

This is lesson 5 of the chapter on watching a model after it goes live.

The data is real and public: the sales of a UK online gift shop from December 2009 to December 2011, published as the Online Retail II dataset. I replay it as if the course's model had gone live in that shop on 1 July 2011, so every date in this lesson is a date in that shop's history, not today's date.

The model is the one from the features chapter's lesson 1. It reads six numbers about each customer, such as how many days since their last order and how much they have spent, and gives each one a score for how likely they are to buy again in the next 30 days. Every night at midnight a job scores every customer the shop knows. I replay 162 nights, from 1 July to 9 December 2011.

Lesson 1 watched the scores with one test and found they barely moved, even when a break hurt the model. Lessons 2 to 4 watched the inputs. In lesson 4 I learned that a drift test's line depends on how much normal wandering the data does. Now I come back to the scores and ask a bigger question: what can the scores tell you on the night they are made, long before anyone knows who really bought?

The survey lesson monitoring and drift detection named this idea in words as watching the predictions. Here I measure it.