Hot Spot

You split data across many servers so no single machine has to do everything. The promise is that with 10 shards, each one handles about a tenth of the load. A hot spot breaks that promise. One shard ends up serving most of the reads and writes while the other nine barely sweat.

The symptom is easy to spot once you look: total throughput is fine on paper, but latency spikes and one node's CPU or disk is pinned at 100 percent while the rest report 10 to 20 percent. Adding more nodes does not help, because the extra capacity lands on the cool shards, not the hot one.

Hot spots come in two flavors. A data hot spot is when one partition holds far more rows than the others. A traffic hot spot is when the rows are spread evenly but one key gets hammered, like the row for a celebrity account or a flash-sale product.

Most systems decide where a key lives by hashing it or by range. With range partitioning, if you key on something monotonic like a timestamp or an auto-increment ID, every new write lands on the same partition, the one holding the newest range. That partition becomes a write hot spot while older partitions go read-only. This is exactly the trap people hit when they use a sequential row key in HBase or Bigtable.

With hash partitioning the data spreads evenly, but a single popular key still maps to a single partition. Hashing user_id does nothing to spread the load of one user_id that gets a million reads a second. The hash sent it all to one place on purpose, so it is reachable.

Low-cardinality keys are another source. If you partition by country and 60 percent of traffic is from one country, that shard carries 60 percent of the load no matter how many shards you have. The key simply does not have enough distinct values to spread work across the cluster.

The first defense is choosing a partition key with high cardinality and even access. Pick something that produces many distinct values and is queried uniformly. If you must key on time, prefix it with a hash or a shard number so writes scatter across partitions instead of stacking on the newest one. This is often called salting the key.

For a single scorching key, you split it. Salting appends a random suffix so one logical key becomes key_0 through key_9 across ten partitions, and a read fans out to all ten. That kills the write hot spot but makes reads more expensive because you now query ten places and merge. Caching the hot key in front of the database is usually cheaper, since a celebrity profile read a million times can be served from Redis instead of the shard.

Some systems rebalance automatically. They detect a partition that is too large or too busy and split it, then move half to another node. DynamoDB adaptive capacity does this by shifting throughput toward the hot partition within seconds. The trade-off is that detection takes time, so you still eat latency during the spike before the system reacts.

Twitter has the classic celebrity problem. A normal user has a few hundred followers, so writing a tweet to every follower's timeline is cheap. A user with 100 million followers would create a write hot spot on whatever shards hold those follower lists every time they post.

Twitter solved it with a hybrid fan-out. For ordinary accounts they push the tweet into each follower's precomputed timeline. For accounts above a follower threshold they skip the push and instead merge the celebrity's tweets in at read time when a follower loads their feed. This avoids the giant write burst on a single popular author while keeping reads fast for everyone else.