A Fix That Looks Like Good Housekeeping
Your retrieval eval, the test that checks whether your search step finds the right document, scores 70%. You open the failures, and something looks odd. In every failed row, the document that came back first has a name very close to the right one. The label says D07, and the retriever returned D07n0. The label says D10, and it returned D10n1.
It is easy to read this as a labelling problem. The thought goes like this: "the index must hold several versions of the same document now, and our golden labels only name one of them. The retriever is fine. The labels are out of date." So somebody writes a small rule: accept the labelled document or any document whose name starts the same way. The eval runs again and reports 100%.
This lesson is about that moment. I ran exactly this situation on the thirty real traces this chapter has used since lesson 85. The strict score was 70%. The "repaired" score was 100%. Then I read the nine rows the rule had turned green. Every single one was a real failure.
So the lesson is not "your strict score is too harsh". It is the opposite. A strict score can be right, and a rule that loosens it can hide exactly the failures you most need to see. I will show you how to tell the difference, and how to loosen a label safely when loosening really is the right thing to do.