How Many Real Answers You Need Before You Can See a Model Get Worse

A basketball coach has a player who has made about 8 of every 10 free throws for two seasons. This week the player seems a little off. Maybe she has slipped to 7 in 10. Maybe it is nothing. The coach does not want to drop a good player because of one bad afternoon, so she decides on a simple rule before she watches: she will only worry if the player scores so few that a real 8-in-10 player would score that few less than 1 time in 20.

Then she counts how often that rule would catch a player who really had slipped to 7 in 10. The answer depends almost entirely on how many shots she watches.

With 10 shots, a player who really slipped gets away with it 85 times in 100. The slip is real, but 10 shots are too few to show it. Only at around 200 shots does the coach catch it nearly every time. A bigger slip, say from 8 in 10 to 4 in 10, would show after a handful of shots. A small one needs many.

Nothing in this table needs a computer to be true; it is plain counting of chances. This lesson asks the same question about a model: how many real answers does a team need before a real drop in the model's quality shows?

This is lesson 7 of the chapter on watching a model after it goes live.

The data is real and public: the sales of a UK online gift shop from December 2009 to December 2011, published as the Online Retail II dataset. I replay it as if the course's model had gone live in that shop on 1 July 2011, so every date in this lesson is a date in that shop's history, not today's date.

The model is the one from the features chapter's lesson 1. It reads six numbers about each customer, such as how many days since their last order and how many orders they have made, and gives each one a score for how likely they are to buy again in the next 30 days. Every night at midnight a job scores every customer the shop knows. Thirty days later the real answer arrives: did that customer buy or not?

Lesson 5 looked at what the scores say before any answer arrives, and lesson 6 at what the first few days of answers say. This lesson asks how many answers it takes. In this shop every customer's answer arrives, about 5,500 a night in the autumn of 2011. Many teams are not that lucky. Some can afford to check only a few hundred cases by hand. Some want to watch one group of customers. On 1 September 2011 the shop's customers came from 41 places, if you count the two labels "Unspecified" and "European Community" as places: 4,849 in the United Kingdom, 90 in Germany and 77 in France, and 38 places with fewer than 50 customers each, Spain with 34 among them. A team that watches one country of this shop, other than the United Kingdom, is watching a small sample.

So the question is the coach's: with a given number of answers, how big must a real drop be before the team can see it?