We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningFoundation · Database Fundamentals
Splitting a database into smaller pieces (shards) distributed across multiple servers. Each shard holds a subset of the data.
In short
Sharding splits one large database into many smaller independent databases called shards, each holding a slice of the rows, so the data and the load spread across many machines instead of one. A shard key decides which shard each row lives on. It is how you scale writes and storage past what a single server can hold, at the cost of much harder cross-shard queries.
In short
Sharding splits one large database into many smaller independent databases called shards, each holding a slice of the rows, so the data and the load spread across many machines instead of one. A shard key decides which shard each row lives on. It is how you scale writes and storage past what a single server can hold, at the cost of much harder cross-shard queries.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Sharding across a run of lessons, not one page. These 4 alone are about 96 minutes of step-by-step reading, every one with a quiz.
Splitting a database across multiple servers, the nuclear option for scaling writes, and the architecture behind Instagram, Uber, and YouTube.
How SQL databases scale across multiple nodes, sharding, query routing, distributed joins, and the coordination challenges.
How a distributed cache spreads keys across nodes with consistent hashing and virtual nodes. Measured tests show why a hot key still overloads a node.
Split massive datasets across multiple nodes so no single machine drowns.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Database Shardingpay once,
yours for life
A single database server has limits: a fixed amount of disk, memory, CPU, and write throughput. You can buy a bigger machine for a while, which is vertical scaling, but eventually the largest available server is not enough, or it gets absurdly expensive. At that point you have to spread the data across many machines, which is horizontal scaling.
Sharding is horizontal scaling for a database. Instead of one server holding all hundred million users, you run several servers and each holds a portion, say twenty-five million each. Reads and writes for a given user go only to the shard that holds that user, so the total capacity is the sum of all the shards. This is the main way large systems scale writes, because unlike read replicas, sharding splits the write load too.
Every sharded system needs a shard key: the column whose value decides which shard a row belongs to. A router computes the shard from the key, for example by hashing the user_id and taking it modulo the number of shards, or by assigning ranges of the key to shards. From then on, any operation that knows the key goes straight to the right shard.
Choosing the shard key well is the hardest and most important decision in sharding. A good key spreads both data and traffic evenly across shards. A bad key creates a hotspot: if you shard an e-commerce system by country and most of your customers are in one country, that one shard gets hammered while the rest sit idle. The infamous version is the celebrity problem, where sharding social data by user means one hugely popular account overwhelms its shard.
Sharding by key
A router sends each row to one shard based on its shard key. Each shard is a full database holding only its slice of the data.
Range sharding assigns contiguous ranges of the key to shards: A to H on shard one, I to P on shard two, and so on. It makes range scans easy, since nearby keys live together, but it is prone to hotspots, because real data is rarely spread evenly across ranges and new sequential keys all land on the last shard.
Hash sharding runs the key through a hash function first and shards on the result. This scatters even sequential or skewed keys evenly across shards, which is great for balancing load, but it destroys locality, so a query for a range of keys now has to hit every shard. Many systems use consistent hashing here so that adding or removing a shard moves only a small fraction of the data rather than reshuffling everything.
In a real system
An application queries through a shard router that maps each shard key to one of several independent databases. No single machine holds all the data.
Sharding buys scale but charges a real tax. A query that does not include the shard key cannot be routed to one shard, so it becomes a scatter-gather that hits every shard and merges the results, which is slow and scales badly. Joins across shards are painful or impossible and often have to be done in the application. Transactions that span shards need distributed-transaction machinery that most teams try hard to avoid.
Resharding, changing the number of shards as you grow, is operationally heavy because it means moving large volumes of live data while the system keeps serving. Because of all this, the usual advice is to delay sharding as long as a single well-tuned database with read replicas can carry the load, and when you do shard, pick a key that keeps the queries you care about on a single shard.
Vitess (YouTube / PlanetScale)
A sharding layer on top of MySQL, built to scale YouTube's database and now the engine behind PlanetScale, hiding the routing from the application.
MongoDB
Has built-in sharding: you pick a shard key and a config server tracks which ranges live on which shards, with automatic balancing.
Citus (PostgreSQL)
An extension that turns PostgreSQL into a distributed, sharded database while keeping standard SQL.
Amazon DynamoDB
Transparently partitions a table by the partition key across many nodes, applying sharding automatically as the table grows.
Sharding splits one database into several smaller independent databases (shards), each holding a subset of the rows, spread across multiple servers. A shard key decides which shard each row lives on. It scales storage and write throughput beyond what a single machine can handle.
A shard key is the column whose value determines which shard a row belongs to. It matters because it controls how evenly data and traffic spread. A good key balances load across shards; a bad key creates a hotspot where one shard gets most of the traffic while others sit idle.
Range sharding assigns contiguous key ranges to shards, which keeps related keys together for easy range scans but is prone to hotspots. Hash sharding shards on the hash of the key, which spreads load evenly but loses locality, so range queries must hit every shard.
Replication copies the same data to multiple servers, mainly to scale reads and survive failures. Sharding splits different data across servers, so each holds only a slice, mainly to scale writes and storage. Large systems usually do both: shard the data, then replicate each shard.
As late as possible. Shard only when a single well-tuned database with read replicas can no longer handle the write volume or data size, because sharding makes cross-shard queries, joins, and transactions much harder. When you do shard, choose a key that keeps your important queries on one shard.
Distributed SQL
How SQL databases scale across multiple nodes, sharding, query routing, distributed joins, and the coordination challenges
Distributed Cache
How a distributed cache spreads keys across nodes with consistent hashing and virtual nodes. Measured tests show why a hot key still overloads a node.
Data Partitioning
Split massive datasets across multiple nodes so no single machine drowns
The RAG Retrieval Cliff: Engineering Recall Back at Scale
The corpus-growth cliff is one equation, lambda equals N times p. Learn the fixes that hold recall at ten million chunks: filtering, partitioning, coarse-to-fine and hierarchical retrieval, ANN tuning, and what each costs.
Replication
Keeping copies of the same data on multiple servers. Improves read performance and provides fault tolerance if one server goes down.
Consistent Hashing
A hashing technique where adding or removing servers only moves a small fraction of keys. Used by Amazon DynamoDB and Cassandra for data distribution.
Horizontal Scaling
Adding more machines to handle increased load (scaling out). Like opening more checkout lanes instead of making one cashier faster.
Database
An organized collection of data that can be easily accessed, managed, and updated. The backbone of almost every application.
SQL
Structured Query Language for managing relational databases. Tables, rows, columns, and powerful joins to query related data.
NoSQL
Databases that don't use traditional table-based relational models. Includes document stores, key-value, graph, and column-family databases.
This page explains the idea. The "Database Sharding" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Database Sharding lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Sharding in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
