We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
6 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Data Replication Distribution
Free preview: the first 2 parts of this 10-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Data Replication Distribution
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Picture a library with ten million books but only one room. People are lined up around the block waiting to check out a single book while the librarian shuffles through an absurd pile. The building is literally sinking under the weight.
That was Instagram in 2012. They had a single Postgres database handling every photo upload, every like, every follow. The thing was groaning under the load. Queries that used to take 5ms started taking 500ms. Their on-call engineers were getting paged at 3 AM because a single table had grown so large that even indexed lookups were slow.
Their fix? They split the data across multiple databases. User IDs 1 through 10 million on one server, 10 million through 20 million on another, and so on. Each database only had to handle a fraction of the total load.
That's data partitioning. You take one giant pile of data and divide it into smaller, more manageable pieces spread across multiple machines. It's one of the most fundamental techniques in distributed systems, and every large-scale system you use today relies on it.
Data partitioning, also called , is the practice of splitting a dataset across multiple storage units so that each unit holds only a subset of the total data.
There are two main flavors:
Horizontal partitioning (sharding) splits rows. If you have a users table with 100 million rows, you might put rows 1-25M on shard 1, 25M-50M on shard 2, and so on. Every shard has the same columns, just different rows.
Vertical partitioning splits columns. You might keep user_id, name, and email in one table, and user_id, bio, avatar_url, and preferences in another. Same rows, different columns. This is useful when some columns are accessed far more frequently than others.
Most people mean horizontal partitioning when they say "partitioning" or "sharding" in a system design context. That's what we'll focus on here.
The key question is always: what's the partition key? That's the field you use to decide which shard a piece of data lands on. Pick the wrong one and you get "hot shards," some machines drowning in traffic while others sit idle.