Caching patterns
“Add a cache” is the most common sentence in system design rounds and the least informative. A cache is four decisions, not one: where it sits, how writes reach it, when entries leave, and what happens on the day it is empty. Get the last one wrong and the cache is the reason you go down rather than the reason you stay up.
Your cache hit ratio slips from 99% to 95%. Traffic has not changed at all. Roughly what happens to the load on your database?
The layers
There is never one cache. By the time a request reaches your code it has passed several, and knowing which layer is responsible for a stale page is most of debugging.
The interesting property of Figure 1 is that each layer is smaller, faster and more stale than the one behind it. Moving a value one layer left makes it faster and more wrong, and that is the only trade you are ever making with a cache.
The number nobody expects
This is the part that surprises people. Hit ratio and database load are not linear: going from 99% to 95% quintuples the misses, because what matters is the miss rate and 5% is five times 1%. A cache tuning change that sounds like a rounding error is a 5x load change on the thing least able to absorb it.
Drag the hit ratio from 99 down to 95 and watch the last row. Four percentage points is a five times increase in database load. This is why “the cache hit rate dipped a bit” is an incident and not a note in a dashboard.
How writes reach the cache
Three patterns, and the choice is about which failure you prefer.
On a write, delete the cache entry rather than writing the new value into it. Two concurrent writers that both update the cache can land in the opposite order to how they landed in the database, and the cache is then permanently wrong with no TTL short enough to make that acceptable. Deleting is idempotent and the next reader repopulates from the source of truth.
The three ways a cache takes you down
Each has a name, a mechanism and a fix. Interviewers ask about the first one constantly.
Stampede, also called the thundering herd. A popular key expires. A thousand concurrent requests all miss, all query the database for the same row at the same instant, and the database falls over serving one value a thousand times. The fix is single flight: the first miss takes a short lock and fetches, everyone else waits for that result. Serving slightly stale data while a refresh is in progress is a cheaper option again.
Penetration. Requests arrive for keys that do not exist anywhere, often from a scanner walking ids. Every one is a miss, so every one hits the database, and the cache provides no protection at all because there is nothing to cache. The fix is to cache the absence: store a null marker with a short TTL. A bloom filter in front is the heavier version for when the key space is genuinely huge.
Avalanche. You populate a cache at deploy time, or a large batch job writes many keys in the same second, and they all carry the same TTL. Some minutes later they all expire at once, and every one of them stampedes together. The fix is one line: add jitter to the TTL, so a nominal ten minutes becomes a random value between nine and eleven.
Break it
Eviction, briefly
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Cache aside | Cache failures degrade latency instead of breaking writes, and only requested data is ever stored. | Every miss pays a round trip, and there is a brief window after a write where a reader can repopulate stale data. | The default. Choose this and justify anything else. |
| Short TTL | Staleness is bounded without any invalidation logic to get wrong. | More misses, so a lower hit ratio and more database load. | Data where a few seconds of staleness is invisible, which is most read paths. |
| Explicit invalidation on write | Near immediate consistency and a high hit ratio at the same time. | Every code path that writes has to remember, and the one that forgets produces a bug nobody can reproduce. | Data users edit and expect to see change immediately, like a profile. |
| Versioned keys | Never invalidate anything. Bump a version in the key and old entries age out on their own. | Wasted memory holding entries nobody will read again, and a version to propagate. | Large derived objects that change wholesale, like a rendered page or a computed feed. |
Interview replay
Checkpoint
1. Hit ratio falls from 99% to 95% with no change in traffic. What happens to database load?
2. A row is updated. Should you write the new value into the cache or delete the key?
3. Your Redis restarts empty during peak traffic. Which defence keeps the site up?
- 99% to 95% hit ratio is five times the database load, not a four percent change.
- A cold cache sends 100% of traffic to the database at once. That is the outage, not the slowdown.
- Delete on write, do not update. A delete is idempotent and a stale write is not.
- Always set a TTL, and jitter it, so a batch of entries written together does not expire together.
A cache is four decisions: where it sits, how writes reach it, when entries leave, and what happens when it is empty. I would default to cache aside, because a cache failure then costs latency rather than breaking writes, and I would delete on write rather than update, since two concurrent updates can land in the cache in the opposite order to the database. TTLs get jitter so a batch of keys does not expire together. The number I would want everyone to know is that database load tracks the miss rate, so a hit ratio slipping from 99 to 95 is five times the load, not four percent more. And the failure I design for is the cache coming back empty at peak: single flight so duplicate misses collapse into one query, plus a concurrency limit in front of the database so overload sheds instead of queues.
