May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
AI Engineering · Serving And Inference
Free preview: the first 2 parts of this 31-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Serving And Inference
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Let me start with a party.
You are hosting a big party, and you want every guest to get a small card with their name and a short note on it. You can do this in two ways.

The first way is the one in the picture. The night before, you sit down and write a card for every person on your list. When a guest walks in, you hand over their card at once. It is fast at the door. But you wrote many cards for people who never came. And if a guest told you some news this morning, the card does not know it, because you wrote it last night.
The second way is to write each card at the door, when the guest arrives. Every card is up to date, and you never write a card nobody uses. But each guest waits a moment while you write, and you must be at the door the whole evening.
A machine learning model that gives each customer a number faces the same choice. In this lesson I measure both ways on a real shop.
This is the first lesson of a new chapter about serving a model: running a trained model so that other programs can ask it questions and get answers.
The model is the one from the features chapter. If it is new to you, please read what a feature is first. In short: a real online shop in the UK sold gifts, mostly to other shops, from 2009 to 2011. For each customer, the model looks at six numbers made from their past orders. Two examples are the days since their last order and how many orders they made. Then it guesses how likely they are to buy again in the next 30 days.
The survey lesson on model serving already explains, in words, the two main ways to serve a model. In batch serving, a job scores many rows on a schedule. In online serving, each request is scored as it comes. It says the main question is whether someone is waiting for the answer. I will not repeat it. Here I measure a different part of the choice: how much an old stored answer costs, how much work a batch wastes, and who it misses.
Two earlier lessons measured something close. Feature freshness found that features up to 14 days old cost nothing measurable on this same shop. And batch vs streaming data found that fresh counts cut the error almost in half for bike rentals in the next hour. This lesson sits between them.