Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Intermediate · LLM and GenAI Ops
Reusing earlier work to avoid paying for the same model call twice: exact caches for identical requests, semantic caches for similar ones, and prompt caching on the provider's side.
In short
LLM caching means reusing earlier work so you do not pay a language model to do the same job twice. An exact cache reuses answers to identical requests, a semantic cache reuses answers to similar questions, and provider prompt caching reuses a repeated start of a prompt.
In short
LLM caching means reusing earlier work so you do not pay a language model to do the same job twice. An exact cache reuses answers to identical requests, a semantic cache reuses answers to similar questions, and provider prompt caching reuses a repeated start of a prompt.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers LLM Caching across a run of lessons, not one page. These 4 alone are about 259 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
RAG triples a chat bill through retrieved context. Cut it with a semantic cache, embedding reuse, and a reranker, without serving confidently wrong answers.
Measured on a laptop with qwen2.5:3b: a 3,951-token prompt took a median 14.35 s to read, and 0.09 s when it was sent again with only the question after it changed. Changing one word 25%, 50% or 75% of the way in made the second read take 0.78, 0.55 and 0.28 of the first, close to 'everything before the first difference is reused'. The reported token count never dropped. Put what changes last.
Semantic search finds text by meaning instead of by shared words. This lesson builds it over this site's 878 lessons and asks 10 questions in six languages. In English the two models were close: 6 and 4 of 10 in the top 5. Across the five other languages, the multilingual bge-m3 found 21 of 50 and the English-trained nomic-embed-text 5 of 50. Four questions were never found. Rewriting one lesson's short description moved it from rank 34-50 to rank 1 in all six languages.
I trained the same boosted trees on the same electricity data 20 times, changing only the seed. Accuracy on the future ran from 0.7179 to 0.7906, 20 different models, a spread of 7.3 points. With one default setting switched off, every seed gave 0.7520. Where the randomness hides in scikit-learn's defaults, what to pin so a run can be repeated, how to log it, and why one run is not enough to compare two models.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
A cache is a store that keeps the result of earlier work, so the next request for the same thing can be answered without doing the work again. Caching is old. What is new with large language models, or LLMs, is how expensive the work is, and how hard it is to tell when two requests are "the same".
An LLM call is billed by the token. A token is a small piece of text, often part of a word. You pay for the input tokens you send and the output tokens the model writes. A call also takes time, often seconds. So every call you can skip saves money and makes the answer faster.
There are three common kinds of LLM caching, and they work at different points on the request path:
The first two skip the model call completely. The third still calls the model, but bills the repeated part at a lower rate.
RAG, retrieval-augmented generation, means searching your own documents and pasting the best matching pieces, called chunks, into the prompt. It stops the model from making things up, but it changes the bill.
A 40 token question becomes a prompt of more than 3,000 tokens, because the chunks travel with it. Those chunks are billed as input tokens on every call. In the lesson's example of a support bot, the retrieved context was about two thirds of the cost of each request. The query embedding, the vector that represents the question, was under one percent.
So the levers that matter are the ones that send fewer context tokens, or skip the call that would send them. A cache does the second. On a hit, the request costs one query embedding instead of a search plus a full generation call.
An exact cache, also called a full-response cache, stores each answer under the exact text of its request. If the same text comes again, it returns the stored answer at zero model cost.
It is never wrong in the sense of answering a different question, because the request matched byte for byte. Its problem is that people rarely type the same thing twice. "How long do I have to request a refund" and "what is the refund time window" are the same question with different words, and an exact cache misses that.
A related cache saves the embedding of a query string you have already turned into a vector. It saves a small cost per hit. It also avoids embedding the same text twice when documents are loaded.
A semantic cache stores each past question as an embedding, a list of numbers that captures its meaning, together with its answer. When a new question arrives, the cache embeds it and searches the stored questions for the nearest one. Questions that mean the same thing land close together, so paraphrases find each other. This is the same idea as semantic search.
The cache then reads the similarity score of the nearest match, often cosine similarity, and compares it with one number, the threshold.
Based on a figure in the lesson RAG Caching and Cost: Semantic Cache, Embedding Reuse, and the Bill.
The saving scales almost directly with the hit rate, because a hit skips the largest cost. The lesson's calculator shows the bill falling in a nearly straight line as the hit rate rises. It stops at a floor made of the embedding lookups and the rented vector index, which no cache can remove.
Ordinary caches fail by serving old data. A semantic cache has a worse failure: it can serve the answer to a different question.
"What is the refund window for digital goods" and "what is the refund window for physical goods" sit very close in embedding space. They have different answers. A loose threshold treats them as the same question, and the user gets the wrong policy, stated with full confidence. Because a hit skips retrieval and generation, nothing later in the pipeline can catch it. This is called a false hit.
The lesson put numbers on it with 12 sample queries, each paired with its nearest cached query.
So the threshold is a correctness setting, not only a cost setting. Start tight, near 0.95. Loosen it only as far as your product can accept, and count false hits on real traffic, because they do not show up in any retrieval metric.
Prompt caching works differently. It does not skip the call. The provider keeps the work it did for the beginning of your prompt, and if your next request starts with exactly the same tokens, it reuses that work. The matched part is billed at a fraction of the normal input price. Only the part after the first difference is billed in full.
The key word is prefix. Caching only covers the tokens up to the first one that differs. So order matters: put stable content first, such as the system prompt, tool descriptions and fixed examples, and put the user's message and the retrieved chunks last. One changing line at the top, such as a timestamp, breaks the match for everything below it.
The providers document their own rules. OpenAI applies a cached-input discount automatically once a prompt is long enough. Anthropic lets you mark which blocks to cache, and charges a small premium to write a block to the cache. Entries expire after a period without use, so caching pays only when traffic keeps them warm. The Token Cost Engineering lesson measures when it pays and when it costs more than it saves.
This connects to the KV cache, which is the cache a model uses inside one request. When the model writes a token, it keeps the key and value vectors it computed for every earlier token, so it does not compute them again for the next token. Prompt caching keeps that kind of work for a repeated prefix across requests.
A semantic cache helps most when traffic repeats and answers are stable: support desks, documentation assistants and internal knowledge bots, which get the same questions in many wordings. Prompt caching helps any workload that resends a long, stable prefix, which is why agents benefit so much.
A semantic cache needs care in three cases. If the answer depends on who is asking, such as their account or region, key the cache on the user plus the question, or do not cache it. If the answer changes over time, such as a price, give entries a short TTL, a time to live. If almost every question is new, the hit rate stays near zero and the lookups are pure overhead.
Last, tie the cache to your documents. When you update a document, answers already in the cache were written from the old version. Expire or version the cache together with the index, or it will keep serving last week's policy. This is the usual cache invalidation problem, with higher stakes.
The RAG Caching and Cost lesson has two calculators to run with your own numbers. One prints the monthly bill with and without a cache. The other shows how a loose threshold serves wrong answers. It is the best next step before you put a semantic cache in front of real users.
GPTCache
An open-source semantic cache for LLM apps: embed the query, search past queries, return the stored answer above a similarity threshold.
Redis
Documents a semantic cache built on its vector search, with a configurable distance cutoff as the main setting.
OpenAI prompt caching
Applies a cached-input discount automatically to long prompts that repeat the same prefix, with no code change.
Anthropic prompt caching
You mark the prompt blocks to cache. Cache reads are billed at a fraction of base input, with a small premium to write.
RAGCache (research)
Caches the model's intermediate states for chunks that are retrieved often, so a repeated chunk is not processed from scratch.
It means a stored result is returned or reused instead of paying the model to do the work again. An exact cache returns the stored answer for an identical request. A semantic cache returns the stored answer for a question with the same meaning. With prompt caching, the provider reuses its work on a repeated prompt prefix and bills those tokens at a lower rate. The model itself does not learn from the cache.
While a model writes an answer one token at a time, it keeps the key and value vectors it already computed for every earlier token. Each new token reuses them instead of recomputing the whole sequence. This KV cache lives inside one request and takes GPU memory. Provider prompt caching extends the idea by keeping that work for a repeated prefix across requests.
A semantic cache is something you run. It matches a new question to a stored one by meaning and, on a hit, skips the model call completely. Prompt caching is run by the provider. The call still happens, but a repeated start of the prompt is billed at a lower rate. They can be used together.
Start tight, around 0.95, where almost every served answer is correct. Loosen it slowly while you count false hits on real traffic. In the lesson's 12-query example, 0.95 served a quarter of the queries with no wrong answers, and 0.85 served three quarters but got three of them wrong.
Caching
Storing frequently accessed data in a faster storage layer so you don't have to fetch it from the original (slower) source every time.
Semantic Search
Search that matches meaning instead of exact words. Text is turned into embeddings, and the results are the documents whose embeddings sit closest to the query's.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
LLM Router
A component that decides which language model should answer each request, so easy requests go to a cheap model and hard ones to a strong model.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "RAG Caching and Cost: Semantic Cache, Embedding Reuse, and the Bill" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the RAG Caching and Cost: Semantic Cache, Embedding Reuse, and the Bill lessoncourse 2
Take models from notebook to production, with labs on real models.