RAG Caching and Cost: Semantic Cache, Embedding Reuse, and the Bill

You measured your chat feature at roughly a tenth of a cent per answer. Then you added retrieval so the model would stop making up policy, and the same answer now costs closer to three tenths. Nothing about the model changed. The question the user typed did not get longer. What changed is that every request now carries a few thousand tokens of retrieved context into the prompt, and you pay for those tokens on every single call.

That is the RAG cost surprise, and it catches teams because the shallow version of this topic talks about the price of an embedding, which is the one number that barely matters. Embedding a query is a fraction of a cent. The bill is somewhere else. It is in the retrieved chunks you stuff into the prompt, which become input tokens metered on the generation call. It is in re-embedding a large corpus every time documents change. It is in the vector store you rent to serve searches at a latency your users tolerate. A RAG request pays several bills a plain LLM call never sees, and if you only optimize the model, you are polishing the smallest line item.

The token cost engineering lesson in the LLMOps chapter covers the parts every LLM call shares: prompt caching of a stable prefix, output caps, batching, resend growth in an agent loop. This lesson is the part that is unique to RAG. Where does the retrieval money go, and what are the levers that move it: caching whole answers by query similarity, reusing embeddings instead of recomputing them, and cutting the context you send. By the end you will have run two calculators, one that prints your monthly bill with and without a cache, and one that shows exactly how a too-loose cache serves wrong answers.