Rate Limits and Provider Failure: The Number One Production Error

The failure that pages you first in an LLM product is almost never a wrong answer. It is 429 Too Many Requests. The model behaves. The bill is fine. What breaks is that real traffic sends requests faster than your provider agreed to serve them, and the provider does exactly what it said it would: it rejects the overflow.

The shape of the incident is always the same. A feature that was smooth in the demo meets launch-day load. A burst of requests trips the per-minute limit, the provider returns 429s, and the code that never expected a 429 does the worst possible thing: it retries immediately, in a tight loop, across every worker at once. Those retries are more load, so they trip the limit harder, so they generate more 429s, so they retry more. The system is now attacking itself. Latency climbs, the queue backs up, and a rate limit that would have shed ten percent of traffic gracefully instead takes the whole feature down.

None of that is a bug in the model, and none of it is fixed by a better prompt. It is an availability problem, and availability is engineering. Rate limits are published contracts. OpenAI documents them in its rate limits guide, and Anthropic documents its own in the rate limits reference. Both enforce limits per minute, on more than one axis, and both expect you to handle rejection as a normal event rather than an exception.

This lesson is the defense. How the limits actually work, why bursty and agentic workloads trip limits that your average load never approaches, and the layered defense (client-side pacing, bulkheads, deadline shedding, jittered backoff, degraded modes, and multi-provider failover) that keeps a 429 from ever reaching a user. The previous lesson on model routing and cascades gave you the fallback models this lesson leans on. This one is about surviving the moment the primary path is throttled or down.