The Reference Window: Compare Tonight With What?

In many homes there is a door frame with small pencil marks on it. Once in a while a parent stands a child against the frame, lays a book flat on her head, and draws a new mark.

Now ask a simple question: has she changed? The answer depends on which mark you compare with.

Compare with last week's mark, and nothing has changed. Compare with the mark from when she was two, and she is far taller. Compare with her brother's mark at the same age, and she is about the same. All three answers are true. They answer three different questions.

Notice one more thing. She grows by herself, every single day. Nobody would call the doctor because she is taller than she was at two. A comparison that always says "she has changed" is true, and useless.

A machine learning model in a live company has the same problem. Every night someone asks "has the data changed?", and the answer depends on what you compare it with. This lesson measures how much.

This is lesson 3 of the chapter on watching a model after it goes live. The model is the same one as before. It reads six numbers about each customer of a real online shop and gives a score from 0 to 1 for how likely they are to buy again in the next 30 days. It went live on 1 July 2011, and a job scores every known customer each night at midnight.

I cannot go back to 2011, so I replay it: I rebuild the table the job would have seen on each night, one night after another, 162 nights in all, from 1 July to 9 December 2011. The six numbers in each row are:

• recency: days since the customer's last order;
• frequency: how many orders they placed that were not returns;
• money: pounds spent, with returns taken off;
• return share: the share of their order lines that were returns, from 0 to 1;
• tenure: days since their first order, how long they have been a customer;
• products: how many different products they have bought.

Two of these, recency and tenure, count days, so they grow by one every night for anyone who does not order again. That is why, in lesson 1, the checks on the model's input rang on every one of those nights, with nothing broken: every customer grew older than anyone in the training rows. In lesson 2 the same thing happened to rules learned from 1, 7, 30 or 91 nights before launch.

Both lessons compared tonight with the past in one way only. Lesson 1 ended with a promise: test better things to compare with, such as last week. This lesson keeps that promise. I keep the same tests and the same lines, and I change only the thing tonight is compared with.

I also build on drift alarms against real harm, which showed on house prices that an alarm for changing data can ring on a harmless change and stay quiet on a harmful one.