May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
AI Engineering · Why Production Breaks
Free preview: the first part of this 23-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Why Production Breaks
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Imagine you want to hire someone to tell you tomorrow's weather. You have a diary that a neighbour kept for two years: every day, the temperature, the rain, the wind. You want to test each person who applies for the job, so you tear out some pages at random and ask them to fill in what those days were like.
One applicant does very well. She opens the diary at a missing Tuesday in October. The rest of that October is still there, so she can see what kind of month it was: warm or cold, wet or dry. She writes something that fits, and she is almost always close. It looks like skill. But she was never asked to do the job you want to hire her for. Nobody asked her about a month she had not seen any of. Tomorrow never comes with the rest of its month already written.

So you change the test. You give her the diary up to some date and ask about the days after it. Now she has only the past to go on, which is all anyone ever has. Her answers get worse. That is not bad news. It is the first honest number you have about her.
This lesson does the same thing to a computer program that guesses how many bikes a city rents in an hour. I tested it the usual way, on hours picked at random, and it looked good. Then I tested it on the hours that came after everything it had learned from, which is the only way it would ever be used. It missed by 1.69 times as much. The rest of the lesson measures where that extra error lives, and what to do so that your own tests tell you the second number before your users do.