Latency

Latency is the elapsed time from the moment you send a request to the moment you get the first useful byte of the response back. It is a duration, not a rate. A page that takes 800 milliseconds to start loading has 800 ms of latency, regardless of how much data follows.

People often confuse latency with throughput and bandwidth. Throughput is how many requests per second you can serve. Bandwidth is how much data fits through the pipe per second. Latency is how long one request takes. A network can have huge bandwidth and still feel slow if every round trip takes 300 ms, because each request waits a long time even though the pipe is wide.

Latency is almost never a single number. Engineers report it as percentiles: p50 (median), p95, p99. A service can have a 50 ms median and a 2 second p99, which means 1 in 100 requests is painfully slow. Averages hide this, so production systems are tuned against tail latency at p99 and p99.9.

A single web request adds up latency at every hop. A DNS lookup costs 10 to 50 ms. Establishing a TCP connection plus a TLS handshake adds one to two more round trips, roughly 50 to 200 ms depending on distance. The server then runs application code, queries a database, and serializes a response before sending it back.

Physical distance sets a hard floor. Light travels about 200 km per millisecond through fiber, so a round trip from London to a server in Virginia is around 80 ms no matter how fast your code is. You cannot optimize your way past the speed of light, you can only move the data closer.

The slowest component dominates the total. If your network adds 5 ms but a database query takes 400 ms because it scans a table without an index, the request is a 400 ms request. This is why profiling matters: you fix the biggest stage first, not the one that is easiest to see.

The main techniques are caching, moving compute closer to users, and cutting round trips. A cache like Redis answers in under 1 ms instead of hitting a 50 ms database query. A CDN such as Cloudflare or CloudFront serves content from an edge location near the user instead of from a single origin thousands of kilometers away. Connection reuse with HTTP keep-alive and HTTP/2 avoids repeating the handshake on every request.

Each fix has a cost. Caching introduces staleness, so you have to decide how out of date the data may be and how to invalidate it. Edge servers multiply your infrastructure footprint and the work of keeping data consistent across regions. Reducing round trips by batching can increase the size and complexity of each payload.

The right target depends on the use case. A trading system fights for microseconds and will colocate servers in the same building as the exchange. A blog is fine at 500 ms. Spending engineering effort to shave 10 ms off a page that already loads in 200 ms is usually wasted unless that page is on a critical path.

When you start a Netflix video, the player does not stream it from one central data center. Netflix places caching servers called Open Connect Appliances directly inside internet service provider networks, so the video data starts from a box that is often in the same city as you. This cuts network latency from tens of milliseconds to single digits and is why playback starts almost instantly.

Compare that to a naive setup where every viewer pulls video from one origin in Virginia. A viewer in Mumbai would eat a round trip of 200 ms or more before a single frame arrived, and tens of millions of viewers would saturate the origin. Moving the bytes physically closer solves both the latency and the throughput problem at once.

The same logic drives Amazon putting warehouses near customers and Google running its own global fiber backbone. The fastest request is the one that has the shortest distance to travel and the least work to do when it arrives.