Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Foundation · LLM and GenAI Ops
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
In short
Retrieval-augmented generation (RAG) answers a question in two steps: a search step finds relevant documents, then a language model reads them in its prompt and answers. It lets a model use current, private documents instead of only its training memory.
In short
Retrieval-augmented generation (RAG) answers a question in two steps: a search step finds relevant documents, then a language model reads them in its prompt and answers. It lets a model use current, private documents instead of only its training memory.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers RAG across a run of lessons, not one page. These 4 alone are about 244 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Why RAG fails in production as the corpus grows: a system that shines on a 100-document demo gets subtly worse at 5 million, offline metrics stay silent, and here are the levers that fight the RAG scale cliff.
Fine-tuning, RAG, and prompting each adapt an LLM differently. Here is what each one changes and a framework for picking the right one.
The operations view of a live RAG plus LLM service: serving, latency budgets, cost per query, caching, index freshness, autoscaling, graceful degradation, retrieval monitoring, guardrails, and SLOs.
The corpus-growth cliff is one equation, lambda equals N times p. Learn the fixes that hold recall at ten million chunks: filtering, partitioning, coarse-to-fine and hierarchical retrieval, ANN tuning, and what each costs.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
A language model answers from what it learned during training. That memory goes out of date, cannot be checked, and does not include your private documents. Retrieval-augmented generation, or RAG, adds a search step. Before the model answers, a retriever finds the documents most related to the question and pastes them into the prompt, and the model answers using them.
The name comes from a 2020 research paper by Patrick Lewis and colleagues, which joined a retriever to a text generator. The pattern is now the usual way to build assistants over company documents, help centres and code, because updating a document is far cheaper than training a model again.
RAG has two halves, and they fail in different ways. Retrieval can miss: the right document never reaches the prompt. Generation can miss: the right document is in the prompt, but the model does not use it, or it adds claims the documents do not support. Every wrong RAG answer comes from one of these two, and they need different fixes.
Ahead of time, documents are cut into chunks of a few hundred tokens (small pieces of text). Each chunk is turned into an embedding, a list of numbers that stands for its meaning, and the embeddings go into a vector index. The same embedding model must be used for documents and questions, because vectors from two different models cannot be compared.
At question time, the question is embedded, and the index returns the nearest chunks, usually by cosine similarity, a score for how closely two embeddings point the same way. Many systems also run a keyword search and merge the two lists. Some then use a reranker, a slower and more accurate model, to reorder the candidates. The top few, a number called k, go into the prompt with an instruction to answer from them.
Then the model writes the answer. Good systems also ask it to cite which document each claim came from, and check afterwards that each claim is supported. That last step is called grounding. It makes the answer auditable: you can follow each sentence back to a source you control.
Retrieval is a competition for a small number of places. The top k does not grow when the collection grows. At a hundred documents, the chunk with the answer has almost no competition. At millions, it has thousands of plausible, on-topic, wrong neighbours, and it must beat all of them to stay in the top k. In a small simulation, recall at 3 (how often the answer chunk is in the top 3) fell from about 0.99 at 20 documents to about 0.5 at 5,000, while the answer chunk itself never moved.
Nothing shows an error when this happens. The query is embedded, the index returns its top k, and the model writes a fluent answer. If the right chunk was not in the top k, the model builds its answer on the wrong material and still sounds sure. Dashboards for latency, errors and throughput stay normal, because the failure is a ranking miss inside a step that succeeded.
Raising k is not a free fix. In a measured lab run over 818 documents and 20 questions, with the small model qwen3:4b asked to pick the right document from the retrieved list, the right document was present for 11 of 20 questions at k of 5 and for 18 of 20 at k of 20. But the questions where the model then picked it fell from 10 of 20 to 6 of 20. With more plausible wrong documents around it, the right one was used less often.
Measure retrieval and generation separately. Recall at k asks whether the right document was in the top k. Faithfulness asks whether each claim in the answer is supported by the retrieved text. A single answer-quality score can tell you the system got worse, but it cannot tell you which half to open.
Measure the floor: the score with no retrieval at all. In a measured run with qwen3:4b, the same 20 questions answered with no documents covered 6.7 percent of the right document's content words, against 15.1 percent with the top 5 documents. So about 44 percent of the reported score was there before retrieval ran. Where retrieval found the right document, the score rose 14.5 points; where it did not, it rose 1.1 points, and on three questions the wrong documents made the answer worse than no documents.
Measure position. With ten retrieved documents and the right one sixth in the list, qwen3:4b picked it for 4 of 14 questions. With the same documents and the same questions, but the right one moved to the last place, it picked it for 14 of 14. The order alone moved the result by ten questions, so your evaluation should record the order of the context.
Use RAG when answers depend on knowledge that changes, is private, or must be traced to a source: product documentation, policies, support articles, internal wikis. To change an answer, you change a document, not the model.
RAG is the wrong tool for teaching a model a new behaviour or a strict output format; that is a job for prompting or fine-tuning. It also adds little where the model already knows the topic. In the measured run above, a question about LoRA scored 20.4 percent with no documents and 22.2 percent with them. For private data the model cannot know, the no-document floor will be near zero, and retrieval does almost all the work.
Start simple and measure. Keep the source text so you can embed it again when you change models. Write twenty questions that each have one known right document, most of them in your users' own words rather than the documents' titles. Then track recall, the end-to-end answer rate and the no-retrieval floor together, and check them again when the collection or the model changes.
LangChain
An open-source framework whose retriever and vector store integrations are widely used to build RAG pipelines.
LlamaIndex
An open-source framework for loading, indexing and querying your own data with a language model, built around the RAG pattern.
pgvector
A PostgreSQL extension that stores embeddings in a vector column and searches them, a common first vector store for RAG.
RAGAS
An open-source evaluation framework that scores RAG systems with metrics such as faithfulness, context precision and answer relevancy.
RAG gives the model new knowledge at question time by putting documents in the prompt, so you update knowledge by updating documents. Fine-tuning changes the model's weights, which suits teaching a behaviour or a format, not facts that change often. Many systems use both.
Either retrieval did not put the right document in the top k, or it did and the model did not use it. Check recall first: pull the retrieved list for the failing question and see if the right document is there. The two cases need different fixes, one in search and one in the prompt or model.
Measure it on your own system. In one measured run, going from 5 to 20 documents raised the chance the right one was present from 11 to 18 of 20 questions, but cut the questions where the model used it from 10 to 6. Choose k from the end-to-end result, not from recall alone.
It reduces them when retrieval finds the right document, but it does not stop them. A model can still add claims the documents do not support, or answer confidently from the wrong documents. A grounding check that compares each claim with the retrieved text is the usual extra layer.
Not always. For a few thousand chunks, exact search in memory or a PostgreSQL extension such as pgvector is enough. Keyword search alone can also be a strong retriever when users type the same words as your documents. A dedicated vector database earns its place at large scale.
Design a RAG System at Scale
System design of a production RAG system. Covers chunking, embeddings, vector search, hybrid retrieval, reranking, caching, evaluation and cost.
Fine-Tuning vs RAG vs Prompting: Choosing Your Approach
Fine-tuning, RAG, and prompting each adapt an LLM differently. Here is what each one changes and a framework for picking the right one.
Production RAG and LLM Systems: Running It, Not Just Building It
The operations view of a live RAG plus LLM service: serving, latency budgets, cost per query, caching, index freshness, autoscaling, graceful degradation, retrieval monitoring, guardrails, and SLOs.
The RAG Retrieval Cliff: Engineering Recall Back at Scale
The corpus-growth cliff is one equation, lambda equals N times p. Learn the fixes that hold recall at ten million chunks: filtering, partitioning, coarse-to-fine and hierarchical retrieval, ANN tuning, and what each costs.
RAG Evaluation: Localizing Failure Between Retrieval and Generation
Recall@k, context precision, and faithfulness split every wrong RAG answer into a retrieval bug or a generation bug, each with a different fix.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
Vector Database
A database that stores embeddings, lists of numbers that capture the meaning of text or images, and finds the ones closest to a question's embedding. It powers semantic search and RAG.
Reranking
A second retrieval stage where a cross-encoder scores each candidate chunk against the query directly. Slower than the first-stage retriever but far more precise, so it runs only over the top few dozen results.
Hallucination
When a language model produces fluent, confident text that is not supported by any source and is often simply false. Retrieval, grounding, and evaluation are the main defenses.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
This page explains the idea. The "The RAG Scale Cliff: What Breaks Between 100 and 5 Million Documents" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the The RAG Scale Cliff: What Breaks Between 100 and 5 Million Documents lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about RAG (Retrieval-Augmented Generation) in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
