Hot keys and hot partitions
Sharding works on an assumption nobody says out loud: load is spread roughly evenly across keys. Hash a million user IDs onto 64 shards and each gets about 1/64 of the traffic. Then Virat Kohli posts, and for the next ten minutes one post ID receives more likes per second than the rest of the table combined. Every one of those writes hashes to the same shard. The other 63 sit idle while one is on fire, and adding shards does nothing, because the problem was never the total. It was one key.
That is a hot key, and the shard it lands on is a hot partition. They show up in every system that shards by key: databases, caches, Kafka topics, rate limiters. Interviewers love them because the usual scaling answer, "add more nodes", is exactly the answer that does not work.
A like counter for each post is one row, sharded by post ID. A celebrity post gets 40,000 likes a second, and one shard handles about 5,000 writes a second. What do you change, and what does it cost you on reads?
Why more shards do not help#
A hash function sends each key to exactly one place. That is what makes lookups fast: to find post 9182, you compute one hash and ask one shard. It also means a single key can never use more than one shard's capacity, however many shards exist.
Every sharded system has a per-partition ceiling, and it is usually much lower than the cluster's:
- DynamoDB: each partition handles at most 3,000 read units and 1,000 write units a second. A table can do millions a second in total, and one key still tops out at 1,000 writes.
- Kafka: a key always goes to the same partition, and a partition is read by one consumer in a group. A hot key means one partition backs up while the other consumers wait for work.
- Redis Cluster: a key lives in one hash slot on one primary. A single Redis primary does on the order of 100,000 simple operations a second on one core, and that is the limit for that key.
- Cassandra and ScyllaDB: a partition key's rows live on the same replicas. A huge or busy partition makes those few nodes slow for every query that touches them.
So the job is to make one logical key look like many physical keys, at least for the operation that is hot. The right way to do that depends on whether the heat is on reads or writes, and on whether the key is predictable (a celebrity account, the home page) or sudden (a post going viral, a flash sale product).
Each write picks one sub-key, so writes spread. Each read of the full value now gathers every sub-key, which is the price.