Tail Latency
The high-percentile response times (p99, p99.9) that affect the slowest requests. A system with 10ms median but 2s p99 latency feels slow for 1 in 100 users.
What is Tail Latency?
In short
Tail latency is the response time of the slowest requests in a system, measured at high percentiles like p99 and p99.9 instead of the average. A service with a 10ms median but a 2 second p99 is fast for most requests yet painfully slow for 1 in every 100, and that slow 1 percent is what users remember and complain about.
What tail latency actually measures
When you sort every request's response time from fastest to slowest, the average sits somewhere in the middle and hides the worst cases. Tail latency looks at the far right end of that sorted list. The p99 is the response time that 99 percent of requests beat, so only the slowest 1 percent are worse. The p99.9 covers the slowest 1 in 1,000, and p99.99 the slowest 1 in 10,000.
These numbers matter because real users do not experience the average. A page that makes 100 backend calls to render will, on average, hit your p99 latency at least once. So if your p99 is 2 seconds, most page loads include a 2 second stall even though your median call is 10ms. The more calls a request fans out to, the more likely it is to touch the tail.
This is why teams that care about user experience set Service Level Objectives on percentiles, not averages. Amazon, Google, and others publish internal targets like p99 under 100ms because they learned that median latency tells you almost nothing about how the product feels.
What causes the tail and how it gets fixed
The tail is made of rare events that pile up. A single request can get unlucky and hit a garbage collection pause, a cold cache miss, a slow disk seek, a noisy neighbor on shared hardware, a lock it has to wait for, a TCP retransmit, or a server that just queued too many requests at once. None of these happen often, but with enough traffic they happen constantly to someone.
Fan-out makes it worse. If one user request triggers 50 parallel backend calls and each backend has a 1 percent chance of a slow response, the odds that at least one of the 50 is slow are about 1 minus 0.99 to the 50th power, roughly 40 percent. So a backend with a great p99 still produces a terrible end-to-end p99 once you fan out.
Common fixes target these tails directly. Hedged requests send a duplicate call to a second replica after a short delay and take whichever returns first, which Google's Jeff Dean described as cutting p99.9 dramatically. Request timeouts with retries cap how bad a single attempt can get. Load shedding rejects excess work before queues build up. Tuning the garbage collector, adding more replicas to reduce queueing, and isolating slow tenants all chip away at the worst percentiles.
When to optimize the tail and the trade-offs
Chasing the tail is expensive, so it is not always the right call. For a batch job that runs overnight, the average throughput matters and a slow request here and there is invisible. For an interactive product, a checkout flow, or any service other services depend on, the tail is the whole game because slow requests stack up across the call graph.
The main trade-off is cost versus consistency. Hedged requests roughly double the load for the requests you hedge, so you trade compute for a tighter p99.9. Adding replicas reduces queueing but costs money and adds coordination. Aggressive timeouts trim the tail but can turn a slow success into a hard failure, so you have to pair them with retries and idempotent operations.
A practical rule: measure first, optimize the percentile your users actually feel, and stop when the SLO is met. Spending weeks to drop p99.99 from 800ms to 600ms is wasted effort if nobody is in that bucket and your p99 is already healthy.
A concrete example
Imagine a product page backed by a microservice that calls a recommendations service. The recommendations service has a median latency of 8ms, a p99 of 40ms, and a p99.9 of 1.5 seconds caused by occasional JVM garbage collection pauses on its servers.
The product page itself looks fast in dashboards because the median is single-digit milliseconds. But the page calls recommendations once per product, and a typical page shows 20 products. The chance that at least one of those 20 calls hits the 1.5 second p99.9 pause is around 2 percent, so roughly 1 in 50 page loads stalls for over a second. Support tickets pile up with vague reports of the site being slow, and nobody can reproduce it because it is random.
The fix is to add a 50ms hedge: if a recommendations call has not returned in 50ms, fire a second call to another replica and use whichever answers first. The duplicate rarely fires, the GC pause on one replica almost never lines up with a pause on the other, and the end-to-end p99.9 drops from 1.5 seconds to under 60ms. Total extra load is only a few percent because hedges only trigger past the 50ms threshold.
Where it is used in production
Coined the modern playbook for tail latency, using hedged requests and tied requests to cut p99.9 across its serving systems.
Amazon
Reported that every 100ms of added latency cost about 1 percent of sales, and built DynamoDB and its services around tight tail latency SLOs.
Netflix
Uses request timeouts, retries, and load shedding via Hystrix and resilience4j to keep one slow dependency from dragging the tail of the whole request.
Cassandra
Offers speculative retry, sending a redundant read to another replica when the first is slow, directly attacking read tail latency.
Frequently asked questions
- What is the difference between average latency and tail latency?
- Average latency is the mean response time across all requests, which a few slow outliers barely move. Tail latency looks at the slowest requests at high percentiles like p99 and p99.9. A system can have a great average and a terrible tail, and users feel the tail.
- What does p99 latency mean exactly?
- p99 is the response time that 99 percent of your requests are faster than. Put another way, only the slowest 1 percent of requests take longer than the p99 value. It is the standard way to describe how bad the slow cases get.
- Why does tail latency get worse with more backend calls?
- If a single user request fans out to many backend calls, it only takes one of them being slow to make the whole request slow. With 100 parallel calls each having a 1 percent chance of hitting the tail, the request is likely to touch a slow call almost every time.
- How do hedged requests reduce tail latency?
- After a short delay, the client sends a duplicate request to a second replica and uses whichever response comes back first. Since the slow events causing the tail are usually independent per server, both copies are rarely slow at once, so the p99.9 drops sharply for only a small increase in total load.
- Should I always optimize for tail latency?
- No. For batch or offline work, throughput and average matter more and rare slow requests are invisible. For interactive, user-facing, or heavily fanned-out services, the tail dominates the experience and is worth optimizing until your percentile SLO is met.
Learn Tail Latency hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Tail Latency as part of a larger topic.
See also
Related glossary terms you might want to look up next.
Latency
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.