Retry

A retry is the simplest fault-tolerance pattern in distributed systems. A call fails, so you call again. The bet is that the failure was transient: a brief network blip, a dropped TCP packet, a server that was busy for 50 milliseconds, or a database that was mid-failover. If you wait a moment and try again, the second attempt often succeeds.

The catch is that not every failure is worth retrying. Retrying a request that returns a 404 Not Found or a 400 Bad Request is pointless, because the answer will be the same every time. Retries only make sense for errors that are expected to clear on their own: connection timeouts, HTTP 503 Service Unavailable, HTTP 429 Too Many Requests, and 5xx server errors.

Equally important, the operation you retry should be safe to repeat. If a payment charge succeeds on the server but the response gets lost, a blind retry could charge the customer twice. That is why retries and idempotency are nearly always discussed together.

A naive retry loop tries again immediately, which is the worst thing to do when a service is overloaded. Every client retrying at once produces a thundering herd that keeps the service down. The fix is exponential backoff: wait longer after each failure. A common schedule is 1 second, then 2, then 4, then 8, doubling each time up to a ceiling.

Backoff alone is not enough, because clients that all failed at the same instant will also all retry at the same instant. Adding jitter (a random amount of delay) spreads those attempts out. AWS recommends full jitter, where the actual wait is a random value between zero and the computed backoff. This smooths the load into a steady trickle instead of synchronized spikes.

Every retry policy needs a hard cap, both on attempts (often 3 to 5) and on total elapsed time. Without a cap, a retry loop can hold a request open for minutes, tying up threads and connections while the user stares at a spinner. Pair retries with a circuit breaker so that once a downstream is clearly dead, calls fail fast instead of retrying into the void.

Use retries for transient, retryable failures on idempotent operations: reading a record, looking up a cache, fetching a config, or any write you have made safe with an idempotency key. They are cheap insurance against the constant low-level flakiness of real networks.

The biggest trap is retry amplification in a chain of services. If service A retries 3 times into B, and B retries 3 times into C, then one user request can become 9 calls hitting C. During an incident this multiplies load exactly when the system can least afford it. A good rule is to retry at only one layer, usually the one closest to the failure, and let other layers fail fast.

The other classic mistake is retrying non-idempotent writes without protection. Sending the same money transfer or order-create twice can cause real damage. The safe pattern is to attach a unique idempotency key to the request so the server recognizes the duplicate and returns the original result instead of doing the work again.

Imagine a checkout service calling a payment API. The first call times out after 2 seconds because the payment provider had a brief hiccup. The client waits a random delay between 0 and 1 second, retries, and this time gets a clean 200 OK. The customer never noticed anything; the retry absorbed the blip.

Now imagine the same call, but the payment actually went through on the provider's side and only the response was lost. Here a plain retry would charge twice. The checkout service avoids this by sending an Idempotency-Key header (a UUID generated once per checkout). The provider sees the same key, knows it already processed that charge, and returns the original successful result. The retry is safe because the write is idempotent.

Stripe's SDK does exactly this: it retries failed requests with exponential backoff and automatically reuses an idempotency key so a retried charge can never bill the customer a second time.