May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
AI Engineering · Serving And Inference
Free preview: the first 2 parts of this 33-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Serving And Inference
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Let me start with a bus.
Imagine you take the same bus to work every morning. Most mornings the ride takes about twenty minutes. On a few mornings it takes twenty-five. And about one morning in a hundred, something odd happens: the driver changes, a road is closed, the doors stick, and the ride takes an hour.

If a friend asks "how long is your ride?", you say "twenty minutes". That answer is true for most days. But it hides the one bad morning, and the bad morning is the one you remember, because it made you late.
Now imagine your day needs ten buses, one after another. Each bus is late only one day in a hundred. How often is at least one of the ten late? If the buses are late for separate reasons, almost one day in ten. If they are late for the same reason, for example snow, they tend to be late on the same days, and the answer changes. Keep that thought: it comes back near the end with real numbers.
A prediction service is like that bus line. Most answers come back quickly. A few come back slowly. In this lesson I send a real service 100,000 requests, five times over. Then I look closely at the slow few: how slow they are, what they have in common, and what happens when a page needs ten answers.
This is lesson 3 of the chapter on serving a model. Lesson 2, latency anatomy, took one request apart and timed every step inside it. It found that a normal request took about one millisecond, and that the model's predict_proba was the biggest step. It looked at the typical request. This lesson looks at the slow ones.
I use the same model, the same service and the same kind of rented machine as lesson 2, so I do not explain them again. In short: the model is the features chapter's gradient boosted tree model. It scores one customer of a real online shop at the start of a month. It scored a test AP of 0.5450.
AP, average precision, is a score from 0 to 1 for how well the model ranks the real buyers above the others. "Test" means it was measured on months the model never trained on. The service is a small web service built with FastAPI and uvicorn, and it reads the customer's six features from a SQLite table. If any of that is new, please read lesson 2 first.
The question for this lesson is simple to ask. Why is the time of the slowest 1 in 100 requests far from the time of the typical one, and what sits in that slow group? Before I measured, I had a confident answer: Python's garbage collector. You will see that the box disagreed.