I/O Optimization

A data engineering team runs a nightly reporting job over a 500 GB dataset on a SATA SSD rated at 550 MB/s sequential throughput. Back-of-napkin math: 500 GB divided by 550 MB/s is roughly 15 minutes. The job takes three and a half hours.

Nobody can explain the gap. The disk is fine. The CPU is idle. Memory is barely used. The job just sits there, grinding through data, and nobody thinks to question the access pattern. The actual throughput is hovering around 40 MB/s, not 550.

That gap between advertised capability and real-world throughput is the entire discipline of I/O optimization. At every company that handles serious data volumes, from LinkedIn sorting Kafka logs to Stripe tuning PostgreSQL block sizes, some engineer eventually sits down and asks: why is the hardware only delivering a fraction of what the spec sheet promises?

The answer is almost always the same. The spec sheet assumes ideal conditions. The workload does not deliver ideal conditions. Fixing that mismatch is the work.

A nightly reporting job over a 500 GB dataset on SATA SSD takes three and a half hours. The drive is rated 550 MB/s sequential, and the dataset divided by that rating says it should finish in 15 minutes. What happened?

Random versus sequential access is the single most misunderstood number in storage engineering. A SATA SSD rated at 550 MB/s sequential delivers around 90,000 random 4KB IOPS, which is 360 MB/s peak under perfect conditions but typically 30 to 50 MB/s on a real workload with queue depth one. Each random IO costs roughly 100 microseconds of latency that the application has to wait through. The drive spec sheet is correct, the engineer's mental model is wrong.

The fix is almost always to sort work to create sequential access. The reporting job sorts input by primary key, the kernel readahead assembles 1 MB transfers, and the drive delivers its full 550 MB/s. Three and a half hours becomes fifteen. Watch the third step, where the same SSD shows three different effective ceilings depending on access pattern.

What This Means in Production

LinkedIn documented a 15x improvement on Kafka log compaction in 2014 when they switched from random seek merge to sorted merge passes. The same principle drives every modern database storage engine: LSM trees in RocksDB and Cassandra exist precisely to convert random writes into sequential ones at the cost of background compaction.

Choice of media matters even more. NVMe drives top 700,000 IOPS per device, SATA SSD around 90,000, spinning disk around 150. Four orders of magnitude span. A workload that fits comfortably on NVMe runs disastrously on HDD even if sequential throughput looks comparable on a spec sheet. AWS exposes IOPS as a separately priced dimension on gp3 and io2 volumes for exactly this reason: if you provision 16,000 IOPS but only need 3,000, you are wasting money.

The three operational moves that come up over and over: sort work to be sequential, batch writes so one fsync amortizes across many records, and pick a block size that matches underlying alignment. Stripe raised PostgreSQL block size from the default 8 KB to 32 KB on analytical replicas where scans dominated and measured a 20 percent throughput gain.

The lesson is that throughput numbers without the access pattern qualifier are useless. Always ask whether the workload is random or sequential before quoting any storage benchmark.