Two Teams Report 75%. Their Next Sprints Share Nothing.
The bottom rungs are within reach and nobody is touching them. That picture is what this lesson is about.
Two teams run a support assistant. Both measure it the same way. Both report the same number: 75% of answers correct, 30 out of 40.
One team spends the next week rewriting prompts. The other spends it changing how documents are cut up and how many get fetched. Only one of them is working on the thing that is actually broken. The 75% cannot tell you which.
This is the gap between having a number and having a diagnosis. One accuracy score only tells you that something is wrong. It does not tell you where.
This lesson closes that gap with one extra field and one extra line of arithmetic.
One word first, because the whole chapter uses it.
A trace is the record of one request, saved at the time it ran. For the system in this lesson, a trace holds four things. The question the user asked, the documents the system fetched, the answer it wrote, and whether that answer was right. Nothing clever. It is a row in a file.
You need traces saved because you cannot do any of this from memory, and you cannot do it from the answer alone. Most of this lesson turns on one field inside the trace, and that field is only there if you wrote it down when the request ran.
Everything below was measured on a real pipeline: a retriever, a generator, 40 questions, and every answer graded by a model, Mistral Large. The answers from the second run were also graded by a model from a different company, Amazon Nova Pro, as a check. The scripts are in scripts/labs/error-analysis, so you can run them and get your own numbers.
The result that drives the whole lesson is this. We changed one setting. The score moved two points, from 73% to 75%, which is nothing. Behind that, the failures completely changed stage. Ten answers were wrong before. Ten were wrong after. They were not the same ten kinds of wrong, and the work to fix them was opposite.