We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
2 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Consistency Models
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Consistency Models
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
When Dropbox first launched, they stored every file every user uploaded, even if 10 users uploaded the exact same 4 GB movie. That was 40 GB for content that could be stored once. At hundreds of petabytes, deduplication saves billions of dollars in storage costs.
Backblaze published storage statistics in 2023 showing that their consumer backup service achieves a 4:1 dedup ratio across all customer data. Their enterprise product, focused on VM images and databases, hits 10-20:1 routinely. At 2 exabytes of logical data, dedup saves them over a billion dollars in raw storage.
Data deduplication identifies redundant copies and stores only one instance, replacing duplicates with references. It works at file level (hash the whole file), block level (hash 4-64 KB chunks), and byte level (find common sequences). The trade-off is CPU for hashing versus storage savings.
Every backup system (Veeam, Commvault, Restic), cloud storage service (S3, Azure Blob), and modern file system (ZFS, btrfs) uses deduplication. Understanding it helps you design storage-efficient systems and explains how Git, layers, and content delivery networks work internally.
The algorithm is simple: hash each data unit, check if that hash exists in the index, and if so, store a pointer instead of the data.
File-level dedup hashes the entire file. If two files match, store one copy. The simplest approach. Catches only exact duplicates.
Block-level dedup divides files into chunks and hashes each independently. Shared sections between files deduplicate even if the files are not identical overall.
Content-defined chunking uses a rolling hash (Rabin fingerprint) to find natural boundaries. This handles insertions better than fixed-size blocks. If you insert 1 byte at the beginning of a file, fixed chunking shifts every boundary and invalidates every chunk. Content-defined chunking only shifts the chunk containing the insertion.
| Level | Granularity | Dedup Ratio | CPU Cost | Best For |
|---|---|---|---|---|
| File | Whole file | 2-3x | Low | Simple file shares |
| Block (fixed) | 4-64 KB | 3-10x | Moderate | Database pages, VM disks |
| Block (variable) | Content-defined | 5-20x | Higher | Backups, general storage |
The hash index problem. At petabyte scale, the dedup hash index itself becomes enormous. With 8 KB average chunks and 32-byte SHA-256 hashes, a petabyte of data produces 125 billion index entries consuming ~4 TB of RAM for the index alone. Production systems use tiered indexes (bloom filter in RAM, full index on SSD) or sampling-based approaches.