We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Microservices Architecture
Automatically re-attempting a failed operation, usually with exponential backoff. Essential for handling transient failures in distributed systems.
In short
A retry is when a system automatically re-attempts an operation that failed, on the assumption the failure was temporary. To avoid hammering a struggling service, retries are usually spaced out with growing delays (exponential backoff) plus a small random jitter, and capped at a fixed number of attempts.
In short
A retry is when a system automatically re-attempts an operation that failed, on the assumption the failure was temporary. To avoid hammering a struggling service, retries are usually spaced out with growing delays (exponential backoff) plus a small random jitter, and capped at a fixed number of attempts.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Retry across a run of lessons, not one page. These 4 alone are about 64 minutes of step-by-step reading, every one with a quiz.
When network calls fail, and they will, retries turn transient failures into invisible hiccups.
Safely retry failed API requests without causing duplicate side effects.
Instead of hammering a failing service, wait longer between each retry, giving the system time to recover.
Add randomness to retry timing to prevent the thundering herd, the missing piece of exponential backoff.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Retry Logicpay once,
yours for life
A retry is the simplest fault-tolerance pattern in distributed systems. A call fails, so you call again. The bet is that the failure was transient: a brief network blip, a dropped TCP packet, a server that was busy for 50 milliseconds, or a database that was mid-failover. If you wait a moment and try again, the second attempt often succeeds.
The catch is that not every failure is worth retrying. Retrying a request that returns a 404 Not Found or a 400 Bad Request is pointless, because the answer will be the same every time. Retries only make sense for errors that are expected to clear on their own: connection timeouts, HTTP 503 Service Unavailable, HTTP 429 Too Many Requests, and 5xx server errors.
Equally important, the operation you retry should be safe to repeat. If a payment charge succeeds on the server but the response gets lost, a blind retry could charge the customer twice. That is why retries and idempotency are nearly always discussed together.
A naive retry loop tries again immediately, which is the worst thing to do when a service is overloaded. Every client retrying at once produces a thundering herd that keeps the service down. The fix is exponential backoff: wait longer after each failure. A common schedule is 1 second, then 2, then 4, then 8, doubling each time up to a ceiling.
Backoff alone is not enough, because clients that all failed at the same instant will also all retry at the same instant. Adding jitter (a random amount of delay) spreads those attempts out. AWS recommends full jitter, where the actual wait is a random value between zero and the computed backoff. This smooths the load into a steady trickle instead of synchronized spikes.
Every retry policy needs a hard cap, both on attempts (often 3 to 5) and on total elapsed time. Without a cap, a retry loop can hold a request open for minutes, tying up threads and connections while the user stares at a spinner. Pair retries with a circuit breaker so that once a downstream is clearly dead, calls fail fast instead of retrying into the void.
Use retries for transient, retryable failures on idempotent operations: reading a record, looking up a cache, fetching a config, or any write you have made safe with an idempotency key. They are cheap insurance against the constant low-level flakiness of real networks.
The biggest trap is retry amplification in a chain of services. If service A retries 3 times into B, and B retries 3 times into C, then one user request can become 9 calls hitting C. During an incident this multiplies load exactly when the system can least afford it. A good rule is to retry at only one layer, usually the one closest to the failure, and let other layers fail fast.
The other classic mistake is retrying non-idempotent writes without protection. Sending the same money transfer or order-create twice can cause real damage. The safe pattern is to attach a unique idempotency key to the request so the server recognizes the duplicate and returns the original result instead of doing the work again.
Imagine a checkout service calling a payment API. The first call times out after 2 seconds because the payment provider had a brief hiccup. The client waits a random delay between 0 and 1 second, retries, and this time gets a clean 200 OK. The customer never noticed anything; the retry absorbed the blip.
Now imagine the same call, but the payment actually went through on the provider's side and only the response was lost. Here a plain retry would charge twice. The checkout service avoids this by sending an Idempotency-Key header (a UUID generated once per checkout). The provider sees the same key, knows it already processed that charge, and returns the original successful result. The retry is safe because the write is idempotent.
Stripe's SDK does exactly this: it retries failed requests with exponential backoff and automatically reuses an idempotency key so a retried charge can never bill the customer a second time.
AWS SDKs
Every official AWS SDK retries throttled and 5xx responses using exponential backoff with full jitter by default.
Stripe
The payment API and SDKs retry failed requests automatically and reuse an idempotency key so a retry can never double-charge.
gRPC
Supports declarative retry policies in service config, including max attempts, backoff, and which status codes are retryable.
Kafka producers
Configurable retries and retry.backoff.ms re-send messages that hit transient broker errors, with idempotent producers preventing duplicates.
A retry re-attempts a single failed call, assuming the problem is brief. A circuit breaker watches the overall failure rate and, once it crosses a threshold, stops calls entirely for a cooldown period so a dead service is not bombarded. They work together: retries handle one-off blips, the breaker stops you from retrying into a service that is fully down.
Without jitter, all clients that failed at the same moment will retry at the same moment, producing synchronized load spikes that can keep a recovering service overloaded. Jitter adds a random delay so the retries spread out into a steady trickle instead of repeated thundering herds.
Retry transient, server-side errors: timeouts, connection resets, HTTP 503, 502, 504, and 429 (with a Retry-After if provided). Do not retry client errors like 400 Bad Request, 401 Unauthorized, 403, or 404, because the result will not change. Be careful retrying 500s, since some are deterministic bugs that will simply fail again.
Most production systems cap at 3 to 5 attempts plus an overall time budget. More attempts give diminishing returns and risk holding connections open too long. Always combine the attempt cap with exponential backoff and jitter, and let a circuit breaker short-circuit once a downstream is clearly unavailable.
If a write succeeds on the server but the response is lost, a retry repeats the write and can charge or create something twice. Prevent it by making the operation idempotent, typically with a unique idempotency key per request. The server recognizes the duplicate key and returns the original result instead of doing the work again.
Idempotency Keys
Safely retry failed API requests without causing duplicate side effects
Exponential Backoff
Instead of hammering a failing service, wait longer between each retry, giving the system time to recover
Jitter
Add randomness to retry timing to prevent the thundering herd, the missing piece of exponential backoff
At-Least-Once Delivery
Never lose a message, but you might see it twice
Poison Message Handling
Detect and isolate messages that crash your consumers, before they crash them forever
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Idempotency
An operation that produces the same result whether you run it once or multiple times. Critical for safe retries in distributed systems.
Bulkhead
A pattern that isolates different parts of a system so a failure in one part doesn't sink the whole ship. Named after the compartments in a ship's hull.
Timeout
A maximum duration to wait for an operation to complete before giving up. Without timeouts, a stalled dependency can hang your entire system indefinitely.
Microservices
An architecture where an application is split into small, independent services that communicate over the network. Each service owns its own data and can be deployed separately.
Monolith
A single, unified application where all features share the same codebase and deployment. Simpler to start with but harder to scale individual parts.
This page explains the idea. The "Retry Logic" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Retry Logic lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Retry in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
