One Wrong Answer, Two Possible Culprits
Your support bot just told a customer the refund window is 30 days. The policy document says 14, and it is sitting right there in the corpus, indexed and searchable, an ordinary chunk among a few hundred thousand others.
There are exactly two stories that produce this answer. In the first, the retriever never surfaced the chunk that states the deadline, so the model filled the gap from its training memory, where some other company's 30 day policy lives. In the second, the retriever delivered the right chunk and the model talked past it anyway. The user sees the same wrong number either way. The fixes have nothing in common. The first story is repaired in the index: chunking, , reranking, the query transform. The second is repaired in the prompt or the model. Pick the wrong story and you can spend a sprint tuning grounding instructions while the actual bug sits in a chunk boundary that split the deadline sentence in half.
This is why a single answer-quality score is useless for debugging. It can tell you the system got worse. It cannot tell you which of the two stages to open, and only has two. That constraint is a gift: with just two suspects and a metric attached to each, every wrong answer becomes a case you can actually prosecute, not a mystery you argue about in a retro. The Evaluating RAG and Agents lesson in the ml-advanced Evals chapter makes the general argument, extending the decomposition to tool calls, trajectories, and task rollups for agents. This lesson stays inside the two RAG stages and works them at the practitioner level: the exact metrics, the diagnostic order, the 2x2 that assigns every failure to a cell with a named fix, and the labeled eval set that makes any of it computable. By the end you should never again look at a wrong RAG answer and shrug.