Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Intermediate · LLM and GenAI Ops
Measuring a retrieval-augmented generation system in two parts, whether retrieval found the right text and whether the answer used it, so a failure can be traced to one of them.
In short
RAG evaluation is measuring a retrieval-augmented generation system in two parts: whether retrieval found the right text, and whether the answer stuck to that text. Scoring the parts separately tells you which stage caused a wrong answer, and so which fix to try.
In short
RAG evaluation is measuring a retrieval-augmented generation system in two parts: whether retrieval found the right text, and whether the answer stuck to that text. Scoring the parts separately tells you which stage caused a wrong answer, and so which fix to try.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers RAG Evaluation across a run of lessons, not one page. These 4 alone are about 262 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Recall@k, context precision, and faithfulness split every wrong RAG answer into a retrieval bug or a generation bug, each with a different fix.
One end-to-end score tells you something broke, not what. Split RAG and agent quality into retrieval, generation, tool, and trajectory evals that localize failures and score the path, not just the answer.
A measured lab where the score moved two points and the diagnosis flipped completely. Ten wrong answers before, ten after, and opposite work to do. The order you fix things in decides whether a sprint lands on the broken stage or the working one.
Generating more eval cases grows the set along the dimensions you described. On a real set, 80% of the difficulty came from near-duplicate documents in the corpus, which no question generator can reach.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
RAG stands for retrieval-augmented generation. A RAG system answers a question in two stages. First, retrieval searches your documents and returns the top few pieces of text, called chunks. Then generation hands those chunks to a language model, which writes the answer.
The lesson starts with a support bot that tells a customer the refund window is 30 days. The policy says 14. Two different stories produce that same wrong answer. In the first, retrieval never returned the chunk with the deadline, so the model filled the gap from its training memory. In the second, the right chunk arrived and the model ignored it.
The fixes for these two stories have nothing in common. The first is fixed in the index: chunking, embeddings, reranking, the query. The second is fixed in the prompt or the model. A single answer-quality score can tell you the system got worse. It cannot tell you which stage to open. RAG evaluation exists to answer that question.
A RAG pipeline has one boundary that matters for measurement. It is the moment the ranked top results leave the retriever and become the model's input. The lesson calls it the seam. That handoff is all the source material the model will ever see for this question.
So retrieval is judged on what crosses the seam: were the right chunks there, and were they ranked high? Generation is judged against what crossed the seam: does the answer stick to that material, and does it answer the question?
This needs one thing in production: logging. Record the retrieved chunk ids and their ranks on every request. Without them, a wrong answer shows that something broke but never where.
Two metrics read the retrieval side. Both need gold labels: a person decides, for each test question, which chunk ids in the corpus are relevant.
Based on a figure in the lesson RAG Evaluation: Localizing Failure Between Retrieval and Generation.
Order matters because the model receives an ordered prompt. The Lost in the Middle paper showed that language models use information at the start and end of a long context better than information in the middle. If retrieval finds the right chunk and then buries it, the usual fix is a reranker, covered in the reranking lesson.
Two classic ranking metrics complete the set. MRR, mean reciprocal rank, is one divided by the rank of the first relevant result. nDCG rewards relevant results for ranking high, with a discount the lower they sit. On one of the lesson's examples, four chunks are relevant, three land in the top 5, and the first is at rank 2. That gives recall@5 of 0.75, MRR of 0.50 and nDCG@5 of 0.59. Report the set, because each one hides a failure the others show.
Raising k lifts recall, because more relevant chunks fit, and lowers precision, because more noise comes too. So treat k as a measured choice, not a default.
Two metrics read the generation side.
One rule surprises people. A claim that is true in the world, but not in the retrieved chunks, counts as unfaithful. That is deliberate. It came from the model's training memory, which goes stale and cannot be checked or corrected by editing a document. Faithfulness is stricter than truth, and that is the point of a RAG system.
Faithfulness is how a RAG evaluation catches hallucination, but only one kind: claims the sources do not back.
Check the metrics in a fixed order. The order is causal, not a matter of style.
Why recall first? A model handed context without the answer will produce unsupported claims whether its grounding is healthy or not. In the refund example, faithfulness was also 0.50, but part of that is the model filling the gap retrieval left. You cannot split the blame until retrieval is fixed and the question is run again.
Give each question two verdicts and it lands in one cell of a 2x2. The tricky cell is bad retrieval with good generation. Faithfulness is high, the dashboard is green, and the answer is wrong, because the model honestly summarized the wrong material.
The lesson's 20-question batch shows how this misleads. Retrieval missed on 8 of 20 questions. Generation failed on 3 of the 12 questions where retrieval worked. Yet average faithfulness was 0.805, because 6 of the 8 retrieval misses produced well grounded answers to the wrong text. The system was retrieval-bound: a 40 percent retrieval failure rate against 25 percent for generation.
None of these metrics can be computed from the system's output alone. Each test question needs a labeled record: the question, the gold chunk ids, and a reference answer written by a person from the source documents. Each run then adds the retrieved ids and the claim verdicts. The human work is done once, and every later run reuses it. This labeled set is often called a golden dataset.
Take questions from real production logs, not ones the team invents. Invented questions tend to be easy and use the documents' own words. The lesson suggests clustering logged questions and sampling across clusters. It also suggests adding some questions with no answer in the corpus, so refusals stay measurable. A set of tens to low hundreds of real, labeled questions is usually enough to locate failures. Freeze it, version it and run it on every change.
One trap: gold chunk ids point at one particular chunking. If you re-cut the documents, as covered under chunking strategies, map the labels again. Otherwise recall keeps printing numbers that mean nothing.
Recall@k and context precision are plain arithmetic once labels exist, so they can run on every commit. Faithfulness and relevance usually need an LLM judge, so check the judge against people. In the lesson's sample, judge and human agreed on 88 of 100 claim verdicts. The 12 disagreements show where the judge or its instructions fail.
The RAG Evaluation lesson has both runnable Python examples, the full decision tree and the 2x2 with its fixes. The Evaluating RAG and Agents lesson extends the same idea to agents.
Ragas
An open-source framework that computes faithfulness, context precision, context recall and answer relevancy, using a language model as the judge.
ARES
A research system that trains small judge models and uses a small human-labeled sample to keep statistical guarantees on the automated RAG scores.
Anthropic Contextual Retrieval
Reports a 49 percent cut in top-20 retrieval failures from contextual embeddings plus contextual BM25, and 67 percent with reranking, with no change to generation.
Measure the two stages apart. For retrieval, check recall@k and context precision against human-labeled gold chunks. For generation, check faithfulness, the share of claims supported by the retrieved text, and answer relevance. Read them in order, recall first, so each wrong answer is traced to one stage.
Retrieval-augmented generation. A search step finds relevant text in your own documents, and a language model writes the answer from that text. Evaluating it means checking both the search step and the answer.
It usually means retrieval failed. Faithfulness only checks that the answer matches the retrieved text, so a model that honestly summarizes the wrong chunks scores high. Check recall against the gold chunks before you trust faithfulness.
Partly. Once people have labeled the relevant chunk ids for each question, recall@k and context precision are simple arithmetic and need no model. Faithfulness and answer relevance need a verdict per claim, which a judge model can give at scale if you check it against human verdicts.
Checked against these on .
Evaluating RAG and Agents: One Score Detects, Components Localize
One end-to-end score tells you something broke, not what. Split RAG and agent quality into retrieval, generation, tool, and trajectory evals that localize failures and score the path, not just the answer.
The Error Hierarchy: Which Failure to Fix First
A measured lab where the score moved two points and the diagnosis flipped completely. Ten wrong answers before, ten after, and opposite work to do. The order you fix things in decides whether a sprint lands on the broken stage or the working one.
Synthetic Cases: A Generator Writes Questions, Not Your Index
Generating more eval cases grows the set along the dimensions you described. On a real set, 80% of the difficulty came from near-duplicate documents in the corpus, which no question generator can reach.
Coverage: The Third of Your Index That Nothing Tests
An eval set is described by how many cases it has. Join it against your index instead: on a real set, 5 of 15 documents were never the right answer to anything, and never appeared in any trace at all.
A Repaired Golden Set Reported 100%. The System Earned 70%.
A strict golden label failed 9 of 30 real traces. A rule that accepts every document named after the label turned 70% into 100%. Reading the nine rows showed every one was a real miss. Loosen labels by reading, never by a rule about names.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
LLM Evaluation
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
LLM-as-Judge
Using a language model to score another model's output. Fast and scalable, but it has real failure modes such as self-preference and position bias, so the judge itself must be validated against human labels.
Chunking Strategies
The rules for splitting documents into pieces before they are embedded for search. Chunk size and boundaries decide what retrieval can find.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
This page explains the idea. The "RAG Evaluation: Localizing Failure Between Retrieval and Generation" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the RAG Evaluation: Localizing Failure Between Retrieval and Generation lessoncourse 2
Take models from notebook to production, with labs on real models.