Your retrieval eval reported 94% in March. It reports 70% now. Nobody shipped a retriever change. The embedding model is the same one. The prompt is the same prompt.

So the team goes looking for the regression.

There is no regression. The system is doing exactly what it did in March. What changed is the ruler.

Open the file. A golden label is one string.

Read the right-hand column carefully, because it is the whole lesson. "D01 is the right answer" is not a fact about the question. It is a fact about the question and everything else the retriever could have returned instead.

You stored the first half. The second half was the state of the index on a Tuesday in March, and nothing wrote it down.

Here is the index those labels were written against, and the index they are being evaluated against now.

Ten pieces of content became twenty-seven documents. The content did not change. The number of places it lives did.

Four ordinary events, two from the documentation side and two from the system side. Not one of them is a bug. Not one of them files a ticket against the eval.

The last one is worth sitting with. A new embedding model does not need to be worse to move your number. It only needs to reorder two documents that were always nearly tied.

For each label, ask a simple question: does the index hold a near-duplicate of the document it names?

All thirty. Not a worrying fraction, not a long tail. Every label in the set names one document in an index that holds more than one copy of it.

Ambiguous is not the same as wrong. D01 is still a correct answer to its question. It has stopped being the only correct answer, and a comparison written as retrieved == gold has no way to know that.

Score the same thirty traces twice. Strict asks whether the retrieved id equals the string in the file. Repaired asks whether the retrieved document carries the content that string was pointing at.

Seventy per cent against a hundred. One run, one system, two readings of the same labels.

Thirty points of reported score, and every one of them is the labels ageing rather than the system slipping.

That is what sends a team after a retrieval bug that does not exist.

"Your labels have drifted" is not something anyone can act on. Nine trace ids are.

Nine rows, three questions, three runs each. The entire repair is three decisions.

Compare that with the alternative. A sprint of retrieval debugging, an embedding comparison, and an argument about rolling back a model nobody changed.

The measurement is one pass over the traces. No model is called and nothing is simulated.

Change TRACES to point at your own run log. The only thing the script needs is, for each case, the id you expected and the id you got.

The repaired number is not automatically the truth, which the next slide is about.

It is tempting to widen every label automatically and move on. Do not.

Two documents can be ninety-nine per cent identical and only one of them acceptable. Last year's runbook and this year's differ by an escalation path. Retrieving the old one is a real failure, and a script that accepts any copy would hide exactly the failure you most need to see.

This is why the lab prints a worklist rather than a corrected score. Nine rows is an afternoon. An automatic repair is a silent second bug layered on the first.

The path from a re-filed runbook to a wasted sprint is four steps, and none of them involves anybody being careless.

Step three is where the information is destroyed. The retriever returned the right content. The comparison had no way to say so, because it was written against a string.

Step four is where it gets expensive.

Every slow failure has the same shape: each step is individually reasonable, and no step is anybody's job to question.

The fourth one is the one to fix first. Your CI checks that the eval ran, that it did not crash, and that the number cleared a threshold. Nothing checks that its answers are still the answers.

This is not a new class of problem, and it is not specific to retrieval. Google wrote it down years ago.

Six months, a two per cent lift from a refresh alone, and nothing on fire the entire time. Their fix is the one a golden set needs too: a number that moves when the thing rots, and somebody who opens it.

The two look the same on a dashboard and they are not the same shape.

A step has a cause you can find in a commit log. A slide has one cause per week, and none of them lives in your repository.

Look at any single week of that slide. It is a point or two, well inside the interval a hundred-case set carries, and it survives review because there is nothing there to see. It only becomes visible once it is large enough that nobody can remember what the index looked like when the labels were written.

The fix is in the schema, not in the score.

Three fields. A set of acceptable ids instead of one. The reason they are acceptable, in words, so the next reader can re-decide. And the date, so the claim can be aged.

The third field is the one that does the most work, and it is the one nobody adds.

Matching on an id feels like the stricter choice. It is not stricter, it is narrower, and the thing it is narrow about is which database row happened to hold the answer.

Most teams re-check golden sets on a calendar reminder, which is the weakest possible trigger.

Three of those four paths end in a re-check, and every one of them is an event your pipeline already emits. The ingest job knows the document count changed. The chunker knows it re-chunked. The deploy that swaps an embedding model knows it swapped one.

A quarterly reminder means the number is wrong for up to a quarter and nobody can say which weeks.

Three of the four steps are code you can write this week. The one that needs a person is small and it is the one that keeps the repair honest.

Neither check needs a new dependency or a model call.

The second snippet is the one to ship first. Two booleans instead of one, and the gap between them on the dashboard next to the pass rate.

That single extra column turns the ambiguous case into a diagnosis. A pass rate falling while the gap grows is label age. A pass rate falling while the gap stays flat is a real regression, and now you know which sprint you are about to spend.

Four lessons ago, the case-difficulty lab reported an overall pass rate of 70% on these same thirty traces. It used that number to talk about how hard the cases were.

Both labs define a pass identically, so the numbers are directly comparable: 21 of 30 in each, checked trace by trace rather than assumed.

Every one of those nine failures was a correct retrieval. The difficulty tiers in that lesson still hold, because they compare cases against each other and the ageing hits them all. The headline rate underneath was carrying nine traces of label age.

That is the uncomfortable part of this lesson. A number you have already read in this course, built from real traces and computed correctly, was still describing the ruler as much as the system.

None of those four questions is about the model, the prompt or the retriever.

A golden set is a measuring instrument, and measuring instruments drift. Yours drifts every time somebody re-publishes a page.

Date your labels. Store a set of acceptable answers rather than one id. Put the strict-to-repaired gap on the dashboard. Then a falling number tells you which of the two things went wrong, which is the only reason to have the number at all.