The Green Dashboard That Meant Nothing Online
Your retrieval change lifted the offline faithfulness score by six points, cleanly above the noise band. The eval gate went green. You shipped it to half of live traffic. Fourteen days later the conversion difference between the two arms was 0.2 points, and the confidence interval ran from slightly negative to slightly positive. The stats tool called it not significant. Nothing had crashed, the offline number was real arithmetic on a real dataset, and yet the metric that pays the team did not move.
This is not a bug and it is not a bad change. It is the most common way engineers new to shipping AI features get burned. They treat a better offline score as a result, when it is only a promise. The offline eval did its job. It told you the change did not regress a metric you already knew how to measure. It never claimed, and cannot claim, that users would behave differently.
The gap between that green dashboard and the flat conversion line is the whole subject of this lesson. Offline evals and online experiments feel like two grades on the same test, so people expect them to agree, and are surprised when they do not. They are not two grades. They answer different questions, they cost different amounts, and a serious team runs both because neither one can do the other's job.