The Oven Light Was Green All Morning
A small bakery opens at six. The baker fills the big mixing bowl from a plain white sack, the same as every morning. The oven light is green. The cakes come out golden and look perfect. The first customers buy them and leave smiling.
That morning the delivery driver left salt in the sugar sack. Nothing in the bakery can see it. The oven works, the mixer works, the cakes look right. The baker finds out only when a customer comes back a few days later, upset, with a half-eaten cake.
Now think about who could have noticed sooner. Someone checking the delivery note when the sack arrived. Someone tasting one cake from the first tray. Or, much later, the customers. Each of them notices a different kind of problem, at a different time.
A machine learning model in a real company lives in the same kind of bakery. This lesson is about who notices when it goes wrong, and how late.
Where This Lesson Starts
This is lesson 1 of the chapter on watching a model after it goes live. In the serving chapter I built a small service around the course's model and measured how fast and how cheaply it answers. That chapter ended with a working service. This chapter asks the next question: once it runs every day, how do I know it is still right?
The model is the one from the features chapter. It reads six numbers about one customer of a real online shop, such as how many days have passed since their last order. It gives back a score between 0 and 1: how likely that customer is to buy again in the next 30 days.
Two earlier lessons already touch this topic, and I build on both. Monitoring and drift detection explains the ideas in words: the kinds of drift, the common tests, and why true answers arrive late. Drift alarms against real harm measured, on house prices, that a drift alarm often rings when nothing is wrong and stays quiet when something is.
Neither lesson looked at time. Here the model runs night after night on the shop's own calendar, and I break it on purpose. The question of this lesson is simple: when something breaks, which watcher notices first, and how many days does it take?