May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
AI Engineering · Serving And Inference
Free preview: the first 2 parts of this 33-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Serving And Inference
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Let me start with a jar.
Imagine a friend asks you the same hard question every few minutes. The first time, you think hard and work out the answer. Then you write it on a small ball and drop it in a jar. The next time the question comes, you just take the ball out. That is much faster than thinking again.

There is a timer next to the jar. You set it when you drop a ball in. When the timer rings, you throw that ball away and think again next time. The timer is how long you trust an old answer.
The jar has one danger. Your answer was right when you wrote it. But maybe something happened since then that changes the answer, and the ball in the jar does not know it.
A model that gives scores can have a jar too. In this lesson I fill one with real requests and measure how often it helps and how often it serves an answer that is out of date.
This is lesson 8 of the chapter on serving a model. The model is the one from lesson 1, batch or online. It reads six numbers about a customer of a real UK gift shop. Then it gives a score: how likely the customer is to buy again in the next 30 days.
Lesson 1 asked a close question. It compared a score made at the moment of a visit with a score made earlier by a nightly, weekly or monthly batch. On this shop, even a score 30.7 days old could not be told apart from a fresh one. A cache is a different machine with a similar risk. A batch scores everyone on a schedule. A cache scores a customer only when they ask, keeps that answer, and gives it out again later. So I wanted to know: does lesson 1's result hold for a cache, and what extra choices does a cache add?
The course has a lesson on caching for retrieval and language models. That one is a semantic cache: it matches two questions that mean the same thing, even when the words differ. This lesson is about something simpler and more common for small models: an exact-key cache, where a hit needs the same key, letter for letter. I do not repeat that lesson here.