We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
4 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Consistency Models
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Consistency Models
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Google processes over 100 petabytes of data per day. At that scale, a 10% reduction in data size saves 10 petabytes of storage daily. That translates to thousands of fewer hard drives, lower electricity bills, less cooling, and faster data transfer between servers.
But compression is not just about saving money. It is about speed. Data that is half the size transfers across the network in half the time. A compressed database page fits more rows in memory, reducing disk I/O. A compressed API response reaches the client faster on a slow mobile connection.
The trade-off is always the same: CPU time to compress and decompress versus storage and savings. Different algorithms make different bets on this spectrum. gzip prioritizes compression ratio. Snappy prioritizes speed. zstd tries to give you both. Brotli squeezes the hardest for web assets at the cost of slow compression.
Understanding compression matters because you encounter it constantly in system design. Kafka messages are compressed. HTTP responses are compressed. Database pages are compressed. Log files are compressed. Parquet files in your are compressed. Every storage and networking decision implicitly involves a compression trade-off.
Cloudflare reported in 2023 that switching from gzip to Brotli for their edge compression reduced bandwidth by 17% across their entire network. At Cloudflare's scale, that is petabytes of savings per month.
Lossless compression means the decompressed data is identical to the original, bit for bit. Used for text, code, database records, and anything where losing even one bit matters.
Lossy compression means some data is permanently discarded during compression. Used for images (JPEG), audio (MP3), and video (H.264) where small losses are imperceptible to humans.
| Algorithm | Type | Speed | Ratio | Use Case |
|---|---|---|---|---|
| gzip/deflate | Lossless | Moderate | Good (~3-5x) | responses, file archives |
| Snappy | Lossless | Very fast | Moderate (~1.5-2x) | , BigTable, real-time streams |
| LZ4 | Lossless | Fastest | Lower (~2x) | In-memory databases, real-time systems |
| zstd | Lossless | Fast | Excellent (~3-8x) | Facebook's choice, best balance |
| Brotli | Lossless | Slow compress, fast decompress | Best for text (~5-10x) | Web assets (HTML, CSS, JS) |
| Run-Length Encoding | Lossless | Trivial | Variable | Simple patterns, bitmaps |
Columnar compression deserves special mention. Column-oriented databases (Redshift, BigQuery, Parquet files) store values from the same column together. Since column values are often similar (thousands of "USA" entries in a country column), compression ratios of 10-20x are common. This is why analytics databases can store petabytes affordably.
Facebook published a benchmark in 2016 comparing zstd against every major algorithm. At compression level 1, zstd matched gzip's ratio while running 4x faster. At level 19, it matched LZMA's ratio while decompressing 10x faster. That benchmark is what convinced the industry to adopt zstd.