We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Microservices Architecture
A maximum duration to wait for an operation to complete before giving up. Without timeouts, a stalled dependency can hang your entire system indefinitely.
In short
A timeout is a hard limit on how long one operation will wait for a response before it gives up and fails. If a network call, database query, or lock acquisition does not finish within the set duration, the caller abandons it and returns an error instead of waiting forever.
In short
A timeout is a hard limit on how long one operation will wait for a response before it gives up and fails. If a network call, database query, or lock acquisition does not finish within the set duration, the caller abandons it and returns an error instead of waiting forever.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Timeout across a run of lessons, not one page. These 4 alone are about 77 minutes of step-by-step reading, every one with a quiz.
Set appropriate timeouts for every external call, deadline propagation and timeout budgets.
A request timeout limits how long a client waits for a full response. Learn how to pick the value, use timeout budgets, and retry without a storm.
Maximum time a client waits to establish a connection before giving up.
Dynamic windows based on activity gaps, group events by user sessions with configurable timeouts.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Timeout Patternspay once,
yours for life
A timeout puts a clock on an operation. You start a request, you set a deadline, and if the result has not arrived by then you stop waiting and treat it as a failure. The operation itself might still be running on the other side, but as far as your code is concerned it is done and you move on.
Without a timeout, a single slow or dead dependency can freeze your service. A thread calls a payment API, the API never answers, and that thread sits blocked. Under load you run out of threads or connections, and a problem in one downstream service turns into a full outage in yours. Timeouts convert an unbounded wait into a bounded, predictable failure you can handle.
There are several kinds you usually set separately. A connection timeout caps how long you wait to establish a TCP connection. A read or socket timeout caps how long you wait for bytes after the connection is open. A request or total timeout caps the whole call end to end. Most outages come from setting one and forgetting the others.
When you make a blocking call, the runtime arms a timer for the deadline you gave it. If the response comes back first, the timer is canceled and you get your data. If the timer fires first, the runtime interrupts the wait, usually by closing the socket or throwing a timeout exception, and the call returns control to you.
In modern systems the deadline is often passed down the call chain. A request comes in with a 2 second budget, the service spends 300ms, then calls a downstream with the remaining 1.7 seconds, which calls its own downstream with what is left. This is deadline propagation, and gRPC supports it directly. It stops a deep chain from each independently waiting its full timeout and blowing past the original budget.
A timeout on its own does not free the work happening downstream. The server may keep processing a query you already gave up on. That is why timeouts pair with cancellation: pass a cancellation token or context so the downstream can see the caller left and stop wasting CPU, connections, and locks.
Picking the number matters. A common rule is to set the timeout above the high percentile of normal latency, often the p99 plus headroom, not the average. If your p99 is 200ms, a 150ms timeout will fail healthy requests, and a 30 second timeout will let real failures hang far too long.
Set timeouts on every call that crosses a process boundary: HTTP requests, database queries, cache lookups, message broker calls, lock acquisition, even DNS. The default in most HTTP clients is no timeout or a very long one, so the dangerous default is doing nothing.
The main trade-off is too short versus too long. Too short and you fail requests that would have succeeded, especially during normal latency spikes, and your retries pile more load onto a service that is already slow. Too long and a stuck dependency drains your thread pool before the timeout ever fires, which defeats the point.
Timeouts rarely work alone. Pair them with bounded retries plus exponential backoff and jitter so a timed-out call gets a second chance without hammering the target. Pair them with a circuit breaker so that once a dependency is clearly down you stop calling it at all instead of timing out on every request. And always give the user a sensible fallback or error rather than a spinner that never resolves.
Picture a checkout service that calls a fraud-scoring API on every order. Normally the API answers in 80ms. One day its database degrades and it starts taking 45 seconds per call. With no timeout, every checkout thread blocks on that call. Within a minute the checkout service exhausts its thread pool, and now nobody can check out, even though the only sick component was fraud scoring.
Now add a 500ms timeout plus a fallback. After 500ms the call is abandoned, the order is queued for asynchronous fraud review, and the customer completes checkout normally. The fraud outage stays contained to fraud scoring. This pattern, fail fast and degrade gracefully, is exactly how a timeout turns a cascading failure into a minor, invisible blip.
Netflix popularized this approach with Hystrix, wrapping every remote call in a timeout, a fallback, and a circuit breaker so one slow service could never take down the whole streaming experience.
A timeout budget is the total time one user request may spend across every attempt and every hop. Retries spend that budget. Three attempts with a 1 second timeout each can wait up to 3 seconds before any backoff is counted, so the per-attempt timeout and the retry count have to fit inside the caller's own timeout. If they do not, the caller gives up while its own retries are still running.
Retries at several layers multiply. Google's SRE book shows three layers that each make four attempts turning one user action into 64 attempts on the database, at the moment the database is already overloaded. Keep retries to one layer where you can, and space them with exponential backoff.
A deadline is a timeout expressed as a point in time that travels with the request. In the SRE book example, server A sets a 30 second deadline, spends 7 seconds, and calls server B with 23 seconds left. gRPC sets no deadline by default, so a client can wait effectively forever unless you set one. When the deadline passes, the client gets DEADLINE_EXCEEDED and the server cancels the call. A distributed job scheduler uses the queue's visibility timeout in a similar way: a job whose worker goes silent comes back and is retried.
Two HTTP status codes carry the word timeout. 504 Gateway Timeout means a proxy or gateway did not get a response in time from the upstream server, for example when NGINX hits its proxy_read_timeout. 408 Request Timeout means the server wants to shut down an unused connection, such as one the client opened and then left idle. The lessons on connection timeouts and request timeouts walk through each layer.
gRPC
Every call carries a deadline that propagates down the chain, so a deep call tree shares one budget instead of stacking timeouts.
Netflix Hystrix and Resilience4j
Wrap remote calls in a timeout plus fallback plus circuit breaker so one slow dependency cannot exhaust threads and cascade.
NGINX
proxy_connect_timeout, proxy_read_timeout, and proxy_send_timeout cap how long the proxy waits on upstream servers before returning a 504.
PostgreSQL
statement_timeout and lock_timeout abort queries or lock waits that run too long, freeing connections and preventing pileups.
Base it on observed latency, not a guess. Set it above your p99 latency with some headroom, so healthy requests almost never trip it but real failures fail fast. If p99 is 200ms, something like 500ms is reasonable. Avoid copying a default like 30 seconds, which is far too long for most internal calls.
A connection timeout limits how long you wait to establish the TCP connection to the server. A read timeout limits how long you wait for data after the connection is open. You should set both, because a server can accept your connection quickly and then stall while processing, and only the read timeout catches that.
Yes, and they are usually used together. The timeout bounds each attempt, and a bounded retry with exponential backoff and jitter gives a timed-out call another chance. Be careful: retrying every timeout immediately can amplify load on an already struggling service, so cap retries and add backoff.
Not by itself. The server may keep running the query or request you gave up on, holding connections and locks. To actually stop downstream work you need cancellation, such as a cancellation token or a gRPC or context deadline that the downstream checks and honors.
A timeout bounds a single call so it cannot hang forever. A circuit breaker watches the recent failure rate across many calls and, once a dependency looks unhealthy, stops sending requests to it entirely for a cooldown period. Timeouts handle one slow call; circuit breakers stop you from making thousands of doomed calls.
A timeout is the longest a program will wait for something before it stops waiting and reports an error. In a distributed system that something is usually a network call, a database query, or a lock. Setting one turns an endless hang into a quick, visible failure your code can handle.
It is a limit placed on every call that crosses a process or network boundary. Designers set separate limits for connecting, for reading the reply, and for the whole request, and they make inner limits shorter than outer ones. Timeouts come up in design reviews and interviews because a missing one lets one slow service take down the services that call it.
A timeout is a duration, such as 2 seconds from now. A deadline is a fixed point in time that is passed down the call chain, so each service knows how much time is left for the whole request. gRPC uses deadlines for this reason, and it sends the remaining time with each call.
A proxy, load balancer, or gateway waited for the server behind it and gave up before a reply came back. The usual causes are a slow or overloaded backend, a query that runs too long, or a proxy timeout set shorter than the backend's normal response time. Look at the upstream service's latency first, then compare it with the proxy's timeout settings.
Start from the time the user or caller can afford to wait. Divide it between attempts, leaving room for backoff between them, and make sure the sum stays under the caller's own timeout. If the numbers do not fit, cut the retry count before you cut the per-attempt timeout below your p99 latency.
Request Timeout
A request timeout limits how long a client waits for a full response. Learn how to pick the value, use timeout budgets, and retry without a storm.
Connection Timeout
Maximum time a client waits to establish a connection before giving up
Timeouts and Fallbacks: What Should a Service Return When Its Lookup Is Slow?
I made the lookup behind a prediction service slow on purpose: 1 lookup in 100 froze for 1 to 2 seconds. Then I tried 21 ways to cope, 5 times each, on a small rented computer with one processor core. With no time limit, the slowest 1 in 100 answers took about 1 second. A 25-thousandths-of-a-second limit (25 ms) with a ready-made backup score brought that down to 26 ms, and about 1.6 answers in 100 were backups. Using the customer's score from last month as the backup hurt the ranking so little I could not measure it. One shared time limit for the whole request kept most of the slowest answers under 25 ms. At 300 requests a second, trying again rescued every frozen lookup. At 450 a second, trying again flooded the store in 3 of 5 runs (94%, 84% and 60% backups), and waiting a random moment first did not help. Every step on the rented computer is recorded, so you can build it yourself.
A Serving Checklist: The Whole Chapter as One Service
I built one small model service twice on a rented computer with one processor core: once as first written, and once with three changes the chapter had measured, a warm-up, one time limit with a stand-in answer, and a batcher. Behind both sat a slow helper that freezes on 1 lookup in 100, so the plain service's failure was built in: it waits for every lookup, and no rate passed. The checked service kept its slowest 1 in 100 answers within 25 ms, with 95 in 100 answers live, up to 450 requests a second in the middle round, about $0.0224 per million answers; the helper's 16 slots, not the processor, set that limit. The lesson ends with the whole chapter as a one-page checklist, and every AWS step is recorded so you can build the machine yourself.
Session Windows
Dynamic windows based on activity gaps, group events by user sessions with configurable timeouts
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Retry
Automatically re-attempting a failed operation, usually with exponential backoff. Essential for handling transient failures in distributed systems.
Latency
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
Microservices
An architecture where an application is split into small, independent services that communicate over the network. Each service owns its own data and can be deployed separately.
Monolith
A single, unified application where all features share the same codebase and deployment. Simpler to start with but harder to scale individual parts.
Service Discovery
The mechanism by which microservices find and communicate with each other. Services register themselves and others can look them up by name.
This page explains the idea. The "Timeout Patterns" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Timeout Patterns lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Timeout in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
