When the Real Answers Come Late: What a Live Model's First Few Days of Results Can Tell You

Imagine you sow a tray of 100 seeds and the packet says "most will come up within a month". You cannot wait a whole month to find out whether this batch is any good. If the seeds are dead, you want to know soon, so you can buy new ones while it is still spring.

So you look every morning. On day 3 a few green tips show. By day 7 more have come up, and by day 14 about half of what will ever come up has done so. You learn to read the early count: "if 10 are up by day 7, that usually means about 35 by the end of the month".

Three things can go wrong with that habit, and each one is worth knowing before you trust it.

First, a few seeds are just slow. On day 3 a slow seed looks exactly like a dead one, although in two weeks it will be a fine plant. An early count mixes up "not yet" with "never".

Second, the days are not all alike. If nobody waters the tray on Saturdays, a count taken on Monday morning after a Friday sowing is low for a reason that has nothing to do with the seeds.

Third, if the whole batch is dead, the early count shows it almost at once. Nothing comes up on day 3, nothing on day 7. You do not need the month to know that.

This lesson does the same thing for a live model whose real answers take 30 days to arrive.

This is lesson 6 of the chapter on watching a model after it goes live.

The data is real and public: the sales of a UK online gift shop from December 2009 to December 2011, published as the Online Retail II dataset. I replay it as if the course's model had gone live in that shop on 1 July 2011, so every date in this lesson is a date in that shop's history, not today's date.

The model is the one from the features chapter's lesson 1. It reads six numbers about each customer, such as how many days since their last order and how much they have spent, and gives each one a score for how likely they are to buy again in the next 30 days. Every night at midnight a job scores every customer the shop knows. I replay 162 nights, from 1 July to 9 December 2011.

The question the model answers has a waiting time built into it. Whether a customer buys "in the next 30 days" is only known when those 30 days are over. Lesson 1 measured what that costs: the watcher that used the real answers noticed a break 31 to 34 days after it started. Lesson 5 asked what the scores alone can say before any answer comes.

This lesson asks the question in between. Some of the answer arrives early. A customer who buys on the second day has already answered "yes". What can those early, partial answers tell a team before the 30 days are up, and how early does a real drop show in them?

The survey lesson monitoring and drift detection named the waiting time in words as the label delay. Here I measure what you can do about it.