May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
AI Engineering · Serving And Inference
Free preview: the first part of this 31-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Serving And Inference
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Let me start at an airport.
Outside the terminal, a small shuttle bus takes people to the hotels. It has 20 seats. The driver has one rule: leave when all 20 seats are full, or 10 minutes after the first person sits down, whichever comes first.

Think about what this rule does. On a busy evening the bus fills in two minutes and leaves. One trip carries 20 people for about the cost of driving one. Without the rule, the driver would make 20 trips, and a long line would form at the door.
On a quiet morning it is different. The first person sits down and waits. Nobody else comes. After 10 minutes the bus leaves with one person in it. That person waited 10 minutes for nothing, and the trip cost the same as a full one.
There is a third way to run the bus. The driver does not wait at all. When the bus comes back, everyone standing at the door gets on, and it leaves at once. On a quiet morning it leaves with one or two people. On a busy evening it leaves full, because people gathered while it was away.
A prediction service can follow the same kinds of rules. Instead of asking the model for one answer at a time, it can collect several requests and ask the model once for all of them. In this lesson I build those rules into a real service and measure both sides: how much work they save, and how long they make people wait.