How Do You Know a New Model Version Did Not Silently Break You?
That is the interview question, asked almost word for word in system design rounds through 2026, and most candidates answer it with monitoring. Monitoring tells you something broke after users hit it. The question is how you know before.
Here is the mechanism the question is really about. You tuned a prompt against a specific model, say gpt-4o-2024-08-06 or claude-3-5-sonnet-20241022. Your few-shot examples, your output format, your refusal handling: all of it is fitted to how that exact snapshot behaves. Then in production you call the model by a floating name, gpt-4o or claude-3-5-sonnet-latest. That name is an alias.
It points at whichever snapshot the provider currently ships, and providers move that pointer. OpenAI documents dated snapshots and an undated name that tracks the current default, and Anthropic publishes a deprecation schedule with dated snapshots you are told to pin against. When the alias rolls forward, your request body does not change. The model answering it does.
The new snapshot is usually better on the provider's own benchmarks and can still be worse on your task. It formats JSON with a stray markdown fence, refuses a class of prompts it used to answer, or shifts tone enough to fail a style check. No exception is raised. looks normal. The 200s keep flowing.
This is what "silently broke you" means. The failure lives inside the content of responses that look correct. So nothing fires: not your error budget, not your dashboards, not your on-call rotation. The break is invisible to the very tools most teams point at reliability. Those tools watch the envelope, the status code and the latency. This failure lives inside the letter.
Notice the asymmetry that makes this so easy to miss. A crash is loud and self-announcing; someone gets paged, a graph turns red, and the fix is prioritized because the signal is undeniable. A behavioral regression is quiet and deniable. The first person to notice is a user who is mildly annoyed, and mild annoyance does not open a ticket. That annoyance has to pile up before anyone files a complaint your team can act on. By then the regression has been shipping for days, against every request. And nobody can say when it started, because nothing recorded the change.
The answer to the interview question is three parts, and the rest of this lesson builds each one. A versioned eval set that you own. A regression gate with a threshold that runs on every prompt edit and every model version bump. And a diff of per-facet pass rates across the two versions. Then the break shows up as a red build with a name on it, not a support ticket a week later.