We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
The high-percentile response times (p99, p99.9) that affect the slowest requests. A system with 10ms median but 2s p99 latency feels slow for 1 in 100 users.
In short
Tail latency is the response time of the slowest requests in a system, measured at high percentiles like p99 and p99.9 instead of the average. A service with a 10ms median but a 2 second p99 is fast for most requests yet painfully slow for 1 in every 100, and that slow 1 percent is what users remember and complain about.
In short
Tail latency is the response time of the slowest requests in a system, measured at high percentiles like p99 and p99.9 instead of the average. A service with a 10ms median but a 2 second p99 is fast for most requests yet painfully slow for 1 in every 100, and that slow 1 percent is what users remember and complain about.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Tail Latency across a run of lessons, not one page. These 4 alone are about 76 minutes of step-by-step reading, every one with a quiz.
P50, P95, P99, measuring the experience of your happiest and most frustrated users.
Segmented eviction, the thing everybody warns about, cost 0.0 points at Guava's default. Lazy maintenance, which nobody mentions, left 5,000 expired entries in memory and one read at 20x the median.
Time delay in system response - most basic performance metric.
Design systems for maximum uptime, eliminate single points of failure across every layer.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Percentilespay once,
yours for life
When you sort every request's response time from fastest to slowest, the average sits somewhere in the middle and hides the worst cases. Tail latency looks at the far right end of that sorted list. The p99 is the response time that 99 percent of requests beat, so only the slowest 1 percent are worse. The p99.9 covers the slowest 1 in 1,000, and p99.99 the slowest 1 in 10,000.
These numbers matter because real users do not experience the average. A page that makes 100 backend calls to render will, on average, hit your p99 latency at least once. So if your p99 is 2 seconds, most page loads include a 2 second stall even though your median call is 10ms. The more calls a request fans out to, the more likely it is to touch the tail.
This is why teams that care about user experience set Service Level Objectives on percentiles, not averages. Amazon, Google, and others publish internal targets like p99 under 100ms because they learned that median latency tells you almost nothing about how the product feels.
The tail is made of rare events that pile up. A single request can get unlucky and hit a garbage collection pause, a cold cache miss, a slow disk seek, a noisy neighbor on shared hardware, a lock it has to wait for, a TCP retransmit, or a server that just queued too many requests at once. None of these happen often, but with enough traffic they happen constantly to someone.
Fan-out makes it worse. If one user request triggers 50 parallel backend calls and each backend has a 1 percent chance of a slow response, the odds that at least one of the 50 is slow are about 1 minus 0.99 to the 50th power, roughly 40 percent. So a backend with a great p99 still produces a terrible end-to-end p99 once you fan out.
Common fixes target these tails directly. Hedged requests send a duplicate call to a second replica after a short delay and take whichever returns first, which Google's Jeff Dean described as cutting p99.9 dramatically. Request timeouts with retries cap how bad a single attempt can get. Load shedding rejects excess work before queues build up. Tuning the garbage collector, adding more replicas to reduce queueing, and isolating slow tenants all chip away at the worst percentiles.
Chasing the tail is expensive, so it is not always the right call. For a batch job that runs overnight, the average throughput matters and a slow request here and there is invisible. For an interactive product, a checkout flow, or any service other services depend on, the tail is the whole game because slow requests stack up across the call graph.
The main trade-off is cost versus consistency. Hedged requests roughly double the load for the requests you hedge, so you trade compute for a tighter p99.9. Adding replicas reduces queueing but costs money and adds coordination. Aggressive timeouts trim the tail but can turn a slow success into a hard failure, so you have to pair them with retries and idempotent operations.
A practical rule: measure first, optimize the percentile your users actually feel, and stop when the SLO is met. Spending weeks to drop p99.99 from 800ms to 600ms is wasted effort if nobody is in that bucket and your p99 is already healthy.
Imagine a product page backed by a microservice that calls a recommendations service. The recommendations service has a median latency of 8ms, a p99 of 40ms, and a p99.9 of 1.5 seconds caused by occasional JVM garbage collection pauses on its servers.
The product page itself looks fast in dashboards because the median is single-digit milliseconds. But the page calls recommendations once per product, and a typical page shows 20 products. The chance that at least one of those 20 calls hits the 1.5 second p99.9 pause is around 2 percent, so roughly 1 in 50 page loads stalls for over a second. Support tickets pile up with vague reports of the site being slow, and nobody can reproduce it because it is random.
The fix is to add a 50ms hedge: if a recommendations call has not returned in 50ms, fire a second call to another replica and use whichever answers first. The duplicate rarely fires, the GC pause on one replica almost never lines up with a pause on the other, and the end-to-end p99.9 drops from 1.5 seconds to under 60ms. Total extra load is only a few percent because hedges only trigger past the 50ms threshold.
Coined the modern playbook for tail latency, using hedged requests and tied requests to cut p99.9 across its serving systems.
Amazon
Reported that every 100ms of added latency cost about 1 percent of sales, and built DynamoDB and its services around tight tail latency SLOs.
Netflix
Uses request timeouts, retries, and load shedding via Hystrix and resilience4j to keep one slow dependency from dragging the tail of the whole request.
Cassandra
Offers speculative retry, sending a redundant read to another replica when the first is slow, directly attacking read tail latency.
Average latency is the mean response time across all requests, which a few slow outliers barely move. Tail latency looks at the slowest requests at high percentiles like p99 and p99.9. A system can have a great average and a terrible tail, and users feel the tail.
p99 is the response time that 99 percent of your requests are faster than. Put another way, only the slowest 1 percent of requests take longer than the p99 value. It is the standard way to describe how bad the slow cases get.
If a single user request fans out to many backend calls, it only takes one of them being slow to make the whole request slow. With 100 parallel calls each having a 1 percent chance of hitting the tail, the request is likely to touch a slow call almost every time.
After a short delay, the client sends a duplicate request to a second replica and uses whichever response comes back first. Since the slow events causing the tail are usually independent per server, both copies are rarely slow at once, so the p99.9 drops sharply for only a small increase in total load.
No. For batch or offline work, throughput and average matter more and rare slow requests are invisible. For interactive, user-facing, or heavily fanned-out services, the tail dominates the experience and is worth optimizing until your percentile SLO is met.
Tail Latency: What Sits in the Slowest 1% of Requests
Why is p99 so far from the median, and what sits in the tail? I sent 100,000 real requests, one at a time, to the features chapter's model on a quiet 1-vCPU EC2 box, five times, and repeated it with four single changes. Here the p99 was only 1.08 times the median and the p99.9 1.23 times, but the slowest request of every run was the very first one, 7.7 times the median. My main guess was wrong: no garbage collection ran during any of the 500,000 measured base requests. Slow requests came in bursts in which every step was slower, and the request after a slow one was slow too 41% to 62% of the time. Because of those bursts, pages of 10 calls in a row met the tail 3.3% to 5.4% of the time, not the 9.56% of 1 - 0.99^10. A warm-up removed the slow first request; gc.freeze() and a higher gc threshold could not be told apart from run-to-run noise. Every AWS step is recorded so you can build the box yourself.
Guava Cache
Segmented eviction, the thing everybody warns about, cost 0.0 points at Guava's default. Lazy maintenance, which nobody mentions, left 5,000 expired entries in memory and one read at 20x the median
Latency
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.
This page explains the idea. The "Percentiles" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Percentiles lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Tail Latency in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
