Can a Model Look Fine for All Customers and Still Be Broken for One Group of Them?

Picture a head teacher in a school with thirty classes. Every week she gets one number: the average mark of every child in the school on that week's test. For years it has sat at about 68 out of 100, and it still does.

What she cannot see in that one number is a class of 25 children whose maths teacher left in September. Their marks have fallen from 68 to 45. But 25 children are a small part of a school of 750, so the school's average falls by less than one mark, and a fall that small happens in plenty of ordinary weeks anyway. The average tells her the school is fine. For those 25 children it is not.

So she decides to look at every class on its own. Now she meets a different problem. A class of 25 can have a bad week for no reason at all: a cold going round, a hard test, a late night before it. Say she calls a class "in trouble" whenever its average is lower than it would be in 95 weeks out of 100 by plain luck. With thirty classes, about four weeks in every five some class will be "in trouble" with nothing wrong anywhere. And the same unlucky class tends to turn up again, because it is the same 25 children week after week.

She also learns something awkward. One class is always a little behind, because it is the class for children who joined the school mid-year. It is "in trouble" every single week. That is true, and it is not news.

This lesson is about all three of her problems, for a model instead of a school.

This is lesson 8 of the chapter on watching a model after it goes live.

The data is real and public: the sales of a UK online gift shop from December 2009 to December 2011, published as the Online Retail II dataset. I replay it as if the course's model had gone live in that shop on 1 July 2011, so every date in this lesson is a date in that shop's history, not today's date.

The model is the one from the features chapter's lesson 1. It reads six numbers about each customer, such as how many days since their last order and how many orders they have placed, and gives each customer a score for how likely they are to buy again in the next 30 days. Every night at midnight a job scores every customer the shop knows. Thirty days later the true answer arrives: did that customer buy or not?

Every lesson so far in this chapter looked at the whole night at once: all the customers' inputs in lessons 2 to 4, all their scores in lesson 5, and all their answers in lesson 6 and lesson 7. Lesson 7 asked how many answers it takes to see a real drop in the whole night. This lesson splits the customers into groups and asks: can the model be fine for the whole night and broken for one group? And what does it cost to watch many groups at once?

The survey lesson monitoring and drift detection described monitoring in words. Here I measure one piece of it, group by group, on the shop's own calendar.