The 200 OK That Lied
Support forwards you a screenshot. Your product told a customer they can return a mattress after 180 nights. The real policy is 30. You open the logs to find out what happened, and the logs have nothing to say: POST /ask, status 200, 950 ms. No exception. No retry. No error budget spent. By every signal your monitoring stack understands, this request went perfectly.
That is the defining property of LLM failures in production, and it breaks the assumption your entire observability toolchain was built on. Classic systems fail loudly: a 500, a timeout, a stack trace. A language model fails by succeeding. It returns a well-formed response, on time, with a plausible token count, and the failure is inside the words. Status codes describe the transport. Nobody told the transport the answer was wrong.
What actually happened to that mattress customer, in the version of this story I have watched more than once: a dependency bump changed the model used at query time, retrieval started returning zero chunks for a slice of queries, the prompt template happily rendered with an empty context block, and the model did what models do with a vacuum. It filled it. Every step returned cleanly. The only artifact that could have shown the empty context block is the exact prompt sent on that exact request, and nobody was saving it.
The Honeycomb team, building one of the first LLM features shipped by an observability company, landed on the same conclusion and wrote it down in All the Hard Stuff Nobody Talks About when Building Products with LLMs: the model is a black box you cannot step through, so the only way to debug it is to capture what went in and what came out, per request, in production. You cannot attach a debugger to a nondeterministic system after the fact. The trace is the debugger.
The previous lesson dealt with the failures that do announce themselves, the 429s and provider outages. This one is about everything that hides behind a 200, and the telemetry that drags it into view.