We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningFoundation · Core Fundamentals
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
In short
Latency is the time delay between sending a request and receiving a response, usually measured in milliseconds. In a web system it includes network travel time, server processing, queue waiting, and database access, and the slowest stage dominates the total.
In short
Latency is the time delay between sending a request and receiving a response, usually measured in milliseconds. In a web system it includes network travel time, server processing, queue waiting, and database access, and the slowest stage dominates the total.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Latency across a run of lessons, not one page. These 4 alone are about 74 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
Time delay in system response - most basic performance metric.
Practical ways to reduce network latency in cloud systems, from region choice and CDNs to protocol tuning. Includes a decision tree to pick a fix.
Automatic memory reclamation, how GC algorithms work and why they cause latency spikes in distributed systems.
Spread data across physical locations to reduce latency and survive regional disasters.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Latencypay once,
yours for life
Latency is the elapsed time from the moment you send a request to the moment you get the first useful byte of the response back. It is a duration, not a rate. A page that takes 800 milliseconds to start loading has 800 ms of latency, regardless of how much data follows.
People often confuse latency with throughput and bandwidth. Throughput is how many requests per second you can serve. Bandwidth is how much data fits through the pipe per second. Latency is how long one request takes. A network can have huge bandwidth and still feel slow if every round trip takes 300 ms, because each request waits a long time even though the pipe is wide.
Latency is almost never a single number. Engineers report it as percentiles: p50 (median), p95, p99. A service can have a 50 ms median and a 2 second p99, which means 1 in 100 requests is painfully slow. Averages hide this, so production systems are tuned against tail latency at p99 and p99.9.
A single web request adds up latency at every hop. A DNS lookup costs 10 to 50 ms. Establishing a TCP connection plus a TLS handshake adds two to three more round trips, roughly 50 to 200 ms depending on distance. The server then runs application code, queries a database, and serializes a response before sending it back.
Physical distance sets a hard floor. Light travels about 200 km per millisecond through fiber, so a round trip from London to a server in Virginia is around 80 ms no matter how fast your code is. You cannot optimize your way past the speed of light, you can only move the data closer.
The slowest component dominates the total. If your network adds 5 ms but a database query takes 400 ms because it scans a table without an index, the request is a 400 ms request. This is why profiling matters: you fix the biggest stage first, not the one that is easiest to see.
The main techniques are caching, moving compute closer to users, and cutting round trips. A cache like Redis answers in under 1 ms instead of hitting a 50 ms database query. A CDN such as Cloudflare or CloudFront serves content from an edge location near the user instead of from a single origin thousands of kilometers away. Connection reuse with HTTP keep-alive and HTTP/2 avoids repeating the handshake on every request.
Each fix has a cost. Caching introduces staleness, so you have to decide how out of date the data may be and how to invalidate it. Edge servers multiply your infrastructure footprint and the work of keeping data consistent across regions. Reducing round trips by batching can increase the size and complexity of each payload.
The right target depends on the use case. A trading system fights for microseconds and will colocate servers in the same building as the exchange. A blog is fine at 500 ms. Spending engineering effort to shave 10 ms off a page that already loads in 200 ms is usually wasted unless that page is on a critical path.
When you start a Netflix video, the player does not stream it from one central data center. Netflix places caching servers called Open Connect Appliances directly inside internet service provider networks, so the video data starts from a box that is often in the same city as you. This cuts network latency from tens of milliseconds to single digits and is why playback starts almost instantly.
Compare that to a naive setup where every viewer pulls video from one origin in Virginia. A viewer in Mumbai would eat a round trip of 200 ms or more before a single frame arrived, and tens of millions of viewers would saturate the origin. Moving the bytes physically closer solves both the latency and the throughput problem at once.
The same logic drives Amazon putting warehouses near customers and Google running its own global fiber backbone. The fastest request is the one that has the shortest distance to travel and the least work to do when it arrives.
Netflix Open Connect
Places video cache servers inside ISP networks so playback starts in single-digit milliseconds instead of crossing continents.
Cloudflare
Runs a global edge network so static content and API responses are served from a location near the user, cutting round-trip time.
Redis
In-memory cache that answers reads in well under 1 ms, replacing slow disk-backed database lookups on hot paths.
Amazon
Found every 100 ms of added latency cost about 1 percent in sales, and uses CloudFront edge caching plus nearby warehouses to keep it low.
Latency is how long one request takes, measured in milliseconds. Throughput is how many requests you can handle per second. A system can have high throughput and still feel slow if each individual request has high latency.
Averages hide the slow requests. A service can have a 50 ms average but a 2 second p99, meaning 1 in 100 requests is painfully slow. Those tail requests hit your heaviest users and break user experience, so production systems are tuned against p95 and p99.
It depends on the use case. Under 100 ms feels instant to a human, 100 to 300 ms is acceptable, and over 1 second makes users leave. A trading system needs microseconds while a blog is fine at 500 ms.
No. Light travels about 200 km per millisecond in fiber, so the round-trip distance sets a hard floor. The only way to beat it is to move the data physically closer to the user with CDNs, edge servers, or caching, not to write faster code.
Usually not. Bandwidth is the width of the pipe; latency is how long a trip takes. Once your pipe is wide enough to carry the data, adding more does nothing for the time each request spends in DNS, handshakes, processing, and travel.
Network Latency Optimization
Practical ways to reduce network latency in cloud systems, from region choice and CDNs to protocol tuning. Includes a decision tree to pick a fix.
Latency Anatomy: Where the Time in One Prediction Request Goes
Where does the time in one prediction request go? I served the features chapter's model with FastAPI, uvicorn and SQLite on a quiet 1-vCPU EC2 box, client and service on one core, and timed every step of 20,000 real requests, five times. At the median, a request took 1,075.9 microseconds. predict_proba on one row took 583.8 of them (54%), and the work outside my own code, HTTP, the framework, the loopback and the client, took 397.0 (37%). The feature lookup, parsing, building the row and writing the answer together took less than 8%. An empty endpoint still cost 333.6 microseconds. Reading the request body had two humps, and a preview of lesson 4 found a one-row predict cost as much as 195 rows inside a 1,000-row call.
Garbage Collection
Automatic memory reclamation, how GC algorithms work and why they cause latency spikes in distributed systems
Geographic Distribution
Spread data across physical locations to reduce latency and survive regional disasters
Global Tables
Fully replicated tables available in every region, low-latency reads worldwide with the trade-off of slower writes
Throughput
The number of operations a system can handle per unit of time. Think of it as how many cars a highway can move per hour.
Bandwidth
The maximum amount of data that can be transferred over a network in a given time. It's the width of the pipe, not how fast the water flows.
CDN
A network of servers distributed globally that caches content close to users. Netflix uses CDNs to stream video from servers near you, not from one central location.
TCP
A reliable transport protocol that guarantees data arrives in order and without errors. It uses a three-way handshake to establish connections.
HTTP
The protocol powering the web. A request-response model where clients ask for resources and servers respond. Stateless by design.
REST API
An architectural style for building APIs using standard HTTP methods (GET, POST, PUT, DELETE). Resources are identified by URLs.
This page explains the idea. The "Latency" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Latency lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Latency in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
