Design a news feed
Reading a feed has to be instant. Nobody waits for a feed. Writing a post can take a minute and nobody would notice, because the person who wrote it already knows what it says.
That asymmetry is the entire design. Everything below is a way of moving work off the read and onto the write, and then dealing with the one case where that stops being possible.
You precompute each user's feed when their friends post. A celebrity with 40 million followers posts. What exactly goes wrong, and would adding servers fix it?
What you are building
- Post something. Text and a media reference. The media pipeline is somebody else’s design question.
- Follow and unfollow. Asymmetric, so a follow is not a friendship.
- Read your feed, newest first, paginated. This is 99% of the traffic and all of the latency budget.
- Feed loads in under 200ms at p95.
- 300 million daily active users, opening the feed about 10 times a day.
- A post shows up for followers within a few seconds, not instantly.
- The feed can be slightly stale. It cannot be empty or out of order.
- Ranking and personalisation. Enormous, and it sits on top of this rather than inside it. Say you would keep the retrieval layer separate from the ranking layer.
- Media upload, transcoding and delivery.
- Notifications, which look similar and are a separate fan out problem.
- Ads insertion, which is a merge step at read time.
The numbers
Reads are 34,722 a second and would be a simple lookup. The fan out turns one post into 200 writes, so the write path is now 10.0 times the read path. That is the trade: you are buying a fast read with a very expensive write, and it works until one user has far more followers than the average.
The API
{
"text": "...",
"mediaId": "m_88123",
"clientId": "c_4f21ab"
}{ "targetId": 88231 }The data model
| post_id | bigint | PK | Snowflake style, so it sorts by time without a secondary index. |
| author_id | bigint | IDX | For the profile page, which is a different query from the feed. |
| text | varchar(2048) | ||
| media_id | varchar(64) | A reference. Never the bytes. | |
| created_at | timestamp |
| user_id | bigint | PK | The reader, not the author. This is the whole point. |
| post_ids | list<bigint> | Newest first, capped at about 800 entries. Ids only, never post content. |
The cap matters. Nobody scrolls past a few hundred items, so trimming the list keeps storage bounded and makes the write cheap. If someone does scroll to the end, fall back to generating older pages on demand, which is rare enough to be slow.
The design
The queue between the post service and the fan out workers is doing more than decoupling. It absorbs the burst when a popular account posts, it lets fan out be retried independently of the post succeeding, and it means the user’s post is durable before any of the expensive work starts.
Deep dive one: push, pull, or both
Deep dive two: why the celebrity breaks it
The fan out model assumes follower counts cluster around an average. They do not. They follow a power law, and a power law has no useful average.
Switch between those two and the trap is obvious. Sizing the system from the first chart gives you a design that the second chart destroys.
Concretely: an account with 40 million followers posts once. That is 40 million list writes from one HTTP request. Adding fan out workers does not fix it, because the work is real, it is not parallelism you are short of, and the last follower gets the post minutes after the first. Meanwhile the fan out queue is full of one person’s post and everybody else’s posts are stuck behind it.
The fix has two halves. Do not fan out above a threshold, and merge those authors in at read time instead. And give large accounts their own queue partition so that when you do fan out to a large but not celebrity account, it does not block everyone else.
“Where do you set the threshold?” Not from first principles. It is where fan out cost crosses read merge cost, and that depends on your read to write ratio and how many celebrities a typical user follows. Start somewhere defensible, around ten to a hundred thousand followers, instrument both paths, and move it. Saying you would measure it is a better answer than a confident number, because a confident number here is invented.
Deep dive three: keeping the feed correct
Precomputed feeds are derived data, and derived data drifts.
Unfollow. Their posts are already sitting in your list. Filtering at read time is the usual answer, since scrubbing the list is expensive and unfollows are rare. Filter on hydration, when you have the author ids anyway.
Deleted posts. Same shape. The id stays in feed lists and hydration returns nothing, so the read path has to tolerate gaps and fetch a few extra to fill the page.
A new follow. The feed has none of that author’s history. Either backfill their recent posts into the list, or accept that the feed fills in going forward. Backfilling is nicer and it is a product decision worth naming.
Rebuilding. Feed lists are a cache with an expensive miss. You need a job that can rebuild one user’s feed from posts and follows, both for corruption and for the day somebody changes the retention rules. If you cannot rebuild it, the derived data is really a source of truth, and now you have two.
Break it
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Fan out on write | A feed read is one list lookup, which is the fastest this can be and scales with cheap read capacity. | One post becomes N writes, and work is wasted on users who never open the app. | The default for the vast majority of accounts, where N is small. |
| Fan out on read | Nothing stored, nothing to rebuild, and no wasted work. Always current. | Every read scatters across hundreds of authors, which is where your latency budget dies. | Celebrity accounts, and users who are rarely active enough to justify precomputation. |
| Storing post ids in feeds | Bounded storage, and editing or deleting a post takes effect everywhere at once. | A second round trip to hydrate, and the read path has to tolerate ids that no longer resolve. | Almost always. Storing content per follower multiplies storage by follower count. |
| Capping feeds at a few hundred entries | Storage and write cost stay bounded no matter how long an account exists. | Deep scrolling falls back to a slower on-demand path. | Always. The number of users who scroll past a few hundred items is small enough to serve slowly. |
Interview replay
Checkpoint
1. Why does adding more fan out workers not solve the celebrity problem?
2. Why store post ids in each feed rather than the post content?
3. A user follows 5,000 accounts and the feed read is now slow. What is the most likely cause?
Reads dominate and nobody waits for a feed, so I would move the work to write time: when someone posts, fan out the post id into a precomputed list for each follower, capped at a few hundred entries. A feed read is then one list lookup plus a batch hydrate of posts by id, and both are cacheable. That model breaks on accounts with huge follower counts, because one post becomes tens of millions of writes and blocks the fan out queue for everyone else, so above a follower threshold I stop fanning out and merge those authors at read time instead. Feed lists store ids only, so edits and deletes take effect in one place, and unfollows and deleted posts are filtered during hydration rather than scrubbed from every list. Ranking sits on top as a separate stage, so it can change independently and degrade to reverse chronological if it fails.
