May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
AI Engineering · Ml Lifecycle
Free preview: the first part of this 26-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Ml Lifecycle
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Imagine a bakery with twenty shops that has sold the same bread for years. The head baker has a new recipe that is cheaper to make. She thinks it tastes just as good, but she is not sure, and a bad loaf in every shop at once would cost her customers. So she has two careful ways to try it.
The first way: sell the new loaf in one shop out of twenty and keep the old loaf everywhere else. Then count the complaints. If the one shop gets more complaints than the other nineteen, the new loaf is probably worse. The trouble is that the shops are different. One shop is near a school and gets fussy children; another is near an office and gets people in a hurry. A few extra complaints in one shop may say more about its customers than about the bread.

The second way: in every shop, bake both loaves for every order. The customer gets the old loaf, as always. A taster in the back tries both loaves from the same order and writes down which one is better. No customer ever eats the new loaf, so no customer is harmed. And because the taster compares two loaves made for the same order, the fussy school children and the busy office workers no longer spoil the comparison. The cost is simple to see: the bakery bakes twice as much bread, and the taster's opinion is not the same as a paying customer's.
Both ways have a delay. A complaint arrives the next day, not at the counter. So the baker has to wait for the answers, and she has to decide how often to look at them. This lesson measures both ways for a program that learns from examples, and asks how long each one takes to notice that the new version is worse.