Cache stampedes
Read firstCaching patterns
The live score of an India vs Pakistan match is cached with a 5 second TTL and read 200,000 times a second. For 5 seconds the database sees nothing. Then the key expires, and every request that arrives in the next 80 milliseconds, while the first one is still rebuilding it, also misses and also queries the database. That is about 16,000 identical queries in one burst, against a database sized for a few hundred. It slows down, the rebuild takes longer, more requests miss in the meantime, and the burst feeds itself.
This is a cache stampede, also called a thundering herd or dogpile. The cache did exactly what it was told. The problem is that a cache with TTLs turns a smooth load into a spike at every expiry, and the spike is largest on exactly the keys that matter most.
A product page for the iPhone launch is cached for 60 seconds and gets 50,000 requests a second. Rebuilding it takes 300 ms of database work. What happens on the database at second 60, and what would you change first?
Why the herd forms#
Two numbers decide how bad it gets: how many requests arrive per second for the key, and how long the rebuild takes. Every request in that window misses.
The herd is bigger than what the database can do in the rebuild window, so the rebuild itself slows down and the window grows. This is how a cache expiry becomes an outage.
The same thing happens at larger scale when many keys expire together: after a cache restart, after a deploy that changed the key format, or when a batch job warmed a million keys with the same TTL an hour ago. That variant is the easiest to fix. Add jitter, a random few percent, to every TTL, so expiries spread out instead of lining up.
const ttl = 300 + Math.floor(Math.random() * 30) // 300 to 329 seconds, never all at onceJitter does nothing for one hot key, though. That needs one of four fixes, and they stack.