May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
AI Engineering · Serving And Inference
Free preview: the first 2 parts of this 26-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Serving And Inference
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Let me start in a coffee shop.
You walk in, order a coffee, and pay. Some minutes later, you have the cup in your hand. The coffee machine itself only ran for a short part of that time. So where did the rest of the wait go?

Many small things happened. The person at the till typed your order. A slip went to the person making drinks. Someone took milk from the fridge. The machine ran. Your name was called, and you walked to the counter. Each step is short. Together they make the wait.
If the shop wants faster coffee, buying a faster machine may not help much. Maybe the till is slow, or the slips wait in a pile. The owner first needs to know which step takes the time. Then they can fix the right one.
A prediction service is like that shop. A request comes in, many small steps happen, and an answer goes out. In this lesson I time every one of those steps, on a real service with a real model, and I find out where the time goes.
This is lesson 2 of the chapter on serving. Lesson 1, batch or online, asked whether to score every customer in a nightly batch or each request as it comes. This lesson takes the second choice, scoring each request as it comes, and looks inside one request.
The model is the one from the features chapter. If it is new to you, please read what a feature is first. It uses six features of each customer of a real online shop, built from their past invoices. It answers one question: at the start of a month, will this customer buy something in the next 30 days? It is a gradient boosted tree model from scikit-learn, and it scored a test AP of 0.5450. AP, average precision, is a score from 0 to 1 for how well the model ranks the buyers above the others.
The survey lesson model serving and inference APIs has a slide called "The Request Path". It walks a request through a gateway, a cache, a and a model, and it gives example times for each hop. I will not repeat that slide here. Instead, I build a small version of that path and measure it.