Network Optimization

A team runs cross-region backups to S3. The uplink is 1 Gbps. The S3 team confirms the bucket can absorb gigabits. The network team confirms the path is clear. And yet the transfer plateaus at 30 Mbps.

Everyone blames everyone else. The application team points at the network. The network team points at S3. S3 points at the application. They are all correct in what they measured and all wrong in what they concluded.

The bottleneck is TCP's default receive window. At 100 ms round-trip time, a 64 KB window caps any single connection at roughly 5 Mbps. Physics does not care about your provisioned bandwidth. The bandwidth-delay product is the hard ceiling, and most engineers never calculate it because the default settings hide the math.

Network optimization is the practice of closing the gap between what the hardware can deliver and what the software actually achieves. The wins here are large, often 5x to 10x, and they compound because every other service on the same path benefits from the same tuning.

A cross region backup to S3 plateaus at 30 Mbps despite a 1 Gbps uplink. The network team swears the link is fine. The S3 team swears the bucket is fine. Both are right. The bottleneck is TCP itself.

The math is unforgiving. Bandwidth delay product for a 100 ms RTT at 1 Gbps is 12.5 megabytes. The default TCP receive window without window scaling is 64 KB, which caps single connection throughput at about 5 Mbps regardless of how fat the pipe is. This calculation has been the starting point of every long-distance throughput investigation since the 1990s. Cloudflare's Marek Majkowski wrote a widely circulated 2018 blog post walking through it in painful detail.

The fix is a four step ladder. Raise tcp_rmem and enable window scaling. Switch congestion control from CUBIC to BBR. Enable GSO and GRO to let the NIC coalesce segments. Add busy polling for latency sensitive flows. Each step adds 2x or 3x and they compose. Watch the third step where BBR alone doubles throughput on a path with 0.1 percent packet loss.

What This Means in Production

Google deployed BBR on YouTube edges in 2016 and measured 4 to 14 percent median throughput gains globally, with much larger improvements on lossy paths. The reason BBR wins is conceptual: CUBIC treats packet loss as congestion and backs off, but on modern internet paths most loss is shallow buffers or radio hops, not actual congestion. BBR models bandwidth and RTT directly and refuses to be fooled.

GSO and GRO offloads let the NIC pack 60 small segments into one large pseudo packet so the kernel processes one structure instead of 60. This alone cuts kernel CPU on a saturated link by 30 to 50 percent. Netflix's open connect appliances reach 800 Gbps per box on FreeBSD using a comparable stack of offloads plus kTLS, which would be impossible with stock socket settings.

Busy polling is the most situational tuning. It reduces interrupt latency on latency sensitive workloads like high frequency trading or telephony but wastes CPU on idle servers. The honest test is to measure interrupt rate and latency before enabling it.

The lesson is that default TCP settings are tuned for safety on the worst possible network, not for performance on the network you actually have. Anyone moving more than a few hundred megabits per second between regions should run through this checklist before assuming the application is at fault.