May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
AI Engineering · Ml Lifecycle
Free preview: the first part of this 24-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Ml Lifecycle
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Imagine a small cafe that has sold the same house coffee for years. A supplier offers a new blend. The owner does not want to change without a good reason, so she runs a taste test: ten regular customers try both cups without knowing which is which, and she will switch if more of them prefer the new one. Six of the ten pick the new blend. Does she switch?
Think about what the six really tells her. If the two blends tasted exactly the same, each customer would be picking at random, like tossing a coin. Six heads out of ten tosses happens all the time. So would five, and so would seven. Ten customers are simply too few to tell a real difference from chance.

There is a second, quieter point in her test. Some customers like both cups equally, or dislike both. They tell her nothing about which blend is better. Only the customers who clearly preferred one cup carry any information. If she has 100 customers and 90 of them cannot tell the cups apart, her decision really rests on the other 10. Asking the same people about both cups, and counting only those who preferred one, is called a paired test: both things are judged on the same cases.
This lesson is about the same decision for a program that learns from examples. A new version of the program wants to replace the one in use. I measured how often three ways of deciding let a new version through when it was no better, and how often they caught one that really was.