We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Distributed Systems Core
The column or field used to determine which shard a piece of data belongs to. A bad shard key creates hot spots; a good one distributes data and queries evenly.
In short
A shard key is the field (or set of fields) a database uses to decide which shard a row or document lives on. The database hashes or ranges that field's value to route every write and read to the correct partition, so the choice of shard key determines whether data and traffic spread evenly or pile up on one overloaded node.
In short
A shard key is the field (or set of fields) a database uses to decide which shard a row or document lives on. The database hashes or ranges that field's value to route every write and read to the correct partition, so the choice of shard key determines whether data and traffic spread evenly or pile up on one overloaded node.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Shard Key across a run of lessons, not one page. These 4 alone are about 109 minutes of step-by-step reading, every one with a quiz.
Splitting a database across multiple servers, the nuclear option for scaling writes, and the architecture behind Instagram, Uber, and YouTube.
Partitioning by contiguous value ranges, the go-to strategy for time-series data, logs, and anything with a natural ordering.
How to choose a cache TTL from your own traffic. A sweep of seven TTL values on skewed traffic shows the hit rate ceiling and the cost of staleness.
Using hash functions to distribute rows evenly across partitions, the antidote to hotspots and skewed data.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Database Shardingpay once,
yours for life
When a single database server can no longer hold all your data or handle all your traffic, you split the data across many servers. Each server holds one slice, called a shard. The shard key is the piece of data the system looks at to decide which slice a given record belongs to.
For example, if you shard a users table by user_id, then user_id is your shard key. Every time you insert or look up a user, the database takes that user_id, runs it through a routing rule, and that rule points to exactly one shard. The record never lives on two shards, and a point query for one user only touches one shard.
A shard key is usually one column, but it can be a compound key made of several columns, such as (tenant_id, created_at). The key is chosen once at design time and is expensive to change later, because changing it means rewriting where every row lives.
There are two common ways to turn a shard key value into a shard. Range-based sharding sorts the key and assigns contiguous ranges to shards. For example user_ids 1 to 1,000,000 go to shard A, 1,000,001 to 2,000,000 go to shard B. This makes range scans efficient but tends to send all the newest inserts to the last shard.
Hash-based sharding runs the key through a hash function and uses the result to pick a shard. A hashed user_id scatters records across all shards evenly, which kills hot spots on writes, but it destroys ordering so range scans now have to touch every shard.
Many systems add a layer of consistent hashing or a lookup table so they can add or remove shards without rehashing every record. MongoDB keeps a config server that maps key ranges (chunks) to shards. Vitess and Citus keep similar routing metadata. The application usually does not pick the shard itself; a router or proxy reads the shard key off the query and forwards it.
The hard rule is that the shard key has to be present in the query for the router to find the data in one hop. A query that filters only on email when the shard key is user_id cannot be routed, so it fans out to every shard and waits for all of them. That scatter-gather query is slow and scales badly.
A good shard key has three properties. It has high cardinality, meaning many distinct values, so data can spread thinly. It has even distribution, so no single value gets a disproportionate share of rows or requests. And it matches your most common query so those queries hit a single shard.
Bad keys create hot spots. Sharding by country puts all of India or the United States on one node. Sharding by a monotonically increasing timestamp or auto-increment id sends every new write to the same shard, so one machine takes all the load while the rest sit idle. Sharding by a low-cardinality status field like active or inactive jams everything into two shards.
The painful trade-off is that one key cannot serve every access pattern. If you shard orders by customer_id, looking up an order by order_id becomes a scatter-gather. Teams handle this with secondary indexes, a global lookup table mapping order_id to shard, or by denormalizing data into a second sharded copy keyed differently. Pick the key that serves your hottest, highest-volume query, and accept that the rarer queries will be slower.
Imagine a multi-tenant SaaS analytics product where each customer (tenant) has millions of events. A natural shard key is tenant_id. Every dashboard query already filters by the logged-in tenant, so those queries route to exactly one shard, and a tenant's data stays physically together.
The risk is a whale tenant. If one customer is 100 times larger than the rest, its single tenant_id value lands all its data on one shard and overloads it while small tenants share another shard with room to spare. The fix is a compound shard key like (tenant_id, event_bucket) that splits a huge tenant across multiple shards while keeping small tenants intact.
This is exactly the kind of decision that is cheap to get right on day one and brutally expensive to fix once you have terabytes spread across the wrong key. That is why shard key selection is treated as an upfront architecture decision, not an implementation detail.
MongoDB
Requires you to pick a shard key per collection; it splits data into chunks by key range and a config server routes queries to the right shard.
DynamoDB
The partition key is its shard key; AWS hashes it to place items, and a poorly chosen key creates a hot partition that throttles throughput.
Cassandra
Uses the first part of the primary key as the partition key, hashed and placed on the ring via consistent hashing to spread data across nodes.
Vitess
Powers YouTube and Slack sharding on top of MySQL; a VIndex maps the shard key column to keyspace shards so queries route to one MySQL instance.
A primary key uniquely identifies a row within a table. A shard key decides which physical shard that row lives on. They can be the same field, but they do not have to be. In Cassandra and DynamoDB the shard (partition) key is only part of the full primary key, and the rest of the key orders rows within that partition.
Not easily. Changing the shard key means almost every row may need to move to a different shard, which usually requires a full data migration into a new cluster keyed differently. Most systems either forbid changing it or make it a heavy operation. Choose carefully up front based on your highest-volume query.
Low cardinality (few distinct values), uneven distribution (one value dominates), or a monotonically increasing value like a timestamp or auto-increment id. All three create hot spots where one shard takes most of the load while others sit idle.
Most likely your queries do not include the shard key, so the router cannot target one shard and has to fan out to all of them (a scatter-gather). The query is then as slow as the slowest shard. Make sure your common queries filter on the shard key, or add a lookup table to translate other identifiers into a shard.
Use a single column when one field has high cardinality and matches your main query. Use a compound key like (tenant_id, bucket) when a single value would be too large for one shard, such as a whale tenant, so you can spread that value across several shards while keeping smaller values together.
Mostly, yes: both name the field that decides where a row lives. MongoDB calls it the shard key, while DynamoDB and Cassandra call it the partition key. In Cassandra the partition key is the first part of the primary key and picks the node; the clustering columns after it only sort rows inside that partition. The data partitioning lesson compares hash and range partitioning, the two usual ways a key becomes a location.
Sharding
Splitting a database into smaller pieces (shards) distributed across multiple servers. Each shard holds a subset of the data.
Consistent Hashing
A hashing technique where adding or removing servers only moves a small fraction of keys. Used by Amazon DynamoDB and Cassandra for data distribution.
Database Partitioning
Dividing a large table into smaller, more manageable pieces while keeping them in the same database. Sharding is partitioning across servers.
CAP Theorem
When a network failure splits a distributed data store, each node must either refuse requests to stay consistent or keep answering with possibly stale data. It cannot guarantee both during the partition.
Consensus
The process of getting multiple nodes in a distributed system to agree on a single value. The foundation of distributed databases and coordination services.
Paxos
A family of protocols for solving consensus in unreliable networks. Famously difficult to understand but mathematically proven correct.
This page explains the idea. The "Database Sharding" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Database Sharding lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Shard Key in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
