35 min read

Design object storage

Staff Files and media Replication Storage and indexesUpdated 28 October 2026

Read firstReplication and quorums Consistent hashing

Object storage is where the internet keeps its files: photos, videos, backups, the raw pages a crawler downloads. You hand it a name and some bytes, and later you ask for the bytes back by that name. Amazon S3 is the famous one.

Amazon publishes a durability figure for S3 of eleven nines, 99.999999999%. In plain words: if you store ten million objects, you should expect to lose one of them roughly once every ten thousand years. Watch one photo get stored, and then watch the machinery that keeps that promise.

Send. A client sends a PUT for photos/2026/08/25/img_88.jpg: the name, and the bytes of the photo.

Disks do not have eleven nines. A typical hard drive has something like a 1 to 2 percent chance of dying in any given year. That sounds small until you own a hundred thousand of them: 1 to 2 percent of 100,000 is 1,000 to 2,000 dead drives a year, several every single day. The whole design is the machinery that turns that hardware into the published number, and the interesting question is not how to store bytes. It is how to make a promise like that and defend it with arithmetic.

A few words first, because the whole page uses them:

An object is one stored file plus a little information about it, like its size and a hash of its contents.
A bucket is a named container of objects, and a key is an object's full name inside it, like 2026/08/25/img_88.jpg. The slashes are just characters. There are no real folders.
Durability is the chance your data still exists. Availability is the chance you can reach it right now. A system that is down for an hour but loses nothing had bad availability and perfect durability.
Metadata is the data about the data: for each object, which machines hold its pieces.

Step 1: Understand the problem#

The number the interviewer cares about is durability, and it is the one candidates skip past fastest. Pin it down first, because it pays for everything else.

You ask	They say	What it settles
How durable, and how available? They are different numbers.	Eleven nines of durability. Four nines of availability.	Everything. Losing data is unacceptable, being briefly unreachable is survivable, so the design spends heavily on spare copies and cheaply on uptime.
How big are objects, and how many?	A few kilobytes to terabytes, and trillions of them.	Two separate problems. Storing the bytes is easy to spread across machines. The trillion small metadata rows are a harder database problem than the storage is.
Can an object be changed in place?	No. Overwrite the whole thing or nothing.	Objects are immutable (never edited after writing). That makes a copy just a copy, makes caching safe, and removes every half-written-update problem you would otherwise design around.
What do reads need to see after a write?	Read after write for new objects.	Once a PUT succeeds, the next GET must find it. That fixes the order of the write path: metadata is committed before the write is acknowledged.
Do users list objects by prefix?	Constantly. It is how everyone uses it.	Metadata has to be kept in key order, not scattered by a hash of the key, and that is what makes the metadata tier hard to split across machines.
One region or many?	Durability inside one region, optional copying to other regions.	Cross region copying is a separate, after-the-fact pipeline. Making every PUT durable in two regions before answering would put an ocean in your write latency.

That fourth answer, "read after write", means any client that reads after a successful PUT sees the new object, never a stale "not found". If the difference between seeing the latest write and catching up later is new, strong vs eventual consistency explains it.

What you are building#

Functional requirements

What the system does.

PUT, GET and DELETE of immutable objects by key. Objects are replaced whole, never edited in place.
Objects from a kilobyte to several terabytes. That range is five orders of magnitude, and it forces two different code paths.
Extreme durability, with the arithmetic to back the claim. This is the product. Availability comes second to not losing data.
Flat key namespace with prefix listing. There are no directories, and pretending otherwise causes the classic hot partition.

Non-functional requirements

How well it has to do it. The numbers you commit to.

Trillions of objects, exabytes in total.
Durability target of eleven nines, defensible with arithmetic.
Read after write for new objects.
A whole datacentre can fail without data loss.

Out of scope

What you cut, and say so out loud.

A normal filesystem interface. Writing into the middle of an object changes everything, which is why this is object storage and not a filesystem.
Access control and encryption, which are real and large but can be designed separately.
Moving old objects to cheaper cold storage, which is a scheduling problem on top of this.

Replication or erasure coding#

This is the decision the whole system turns on. Start with the obvious answers and see where each one breaks.

How do you keep a file safe when disks keep dying?

First try
One copy on a good disk
Write the object to one disk and trust it.
Where it breaks: With a 1 to 2 percent yearly failure rate per disk, a fleet of 100,000 disks loses several every day, and every object on them is simply gone.
Better
Three full copies on three machines
Write the whole object three times. A read can use any copy, and losing two copies is still fine.
Where it breaks: It costs 3 GB of disk for every 1 GB stored. At exabyte scale that bill is the business, and it still survives only two failures.
What you ship
Erasure coding for big objects, copies for small ones
Cut the object into 6 data fragments, compute 3 parity fragments, and put all 9 on different machines. Any 6 rebuild it, so it survives 3 failures while storing 1.5 GB per 1 GB. Tiny objects still get 3 copies, because cutting a 4KB object into nine pieces costs more than it saves.

Parity is the trick that makes the third rung work, and the smallest version fits in your head. Take two data fragments, A = 5 and B = 3, written in bits as 0101 and 0011. Store a third fragment P = A XOR B (XOR sets a bit to 1 where the two inputs differ), which is 0110, or 6. Now lose B. XOR the survivors, A and P: 0101 XOR 0110 = 0011, which is 3, and B is back. That is one parity fragment covering one loss. The codes real systems use extend the same idea so that 3 parity fragments can stand in for any 3 lost fragments out of 9.

Here are the two schemes side by side, for a 1 GB object.

Durability scheme

Three complete copies. Simple, and a read is a read from any one of them, so reads are cheap and a read with a copy missing costs nothing extra. It survives two failures and costs 3 GB to store 1 GB. For small objects this is the right answer, because the fixed cost of splitting a 4KB object into fragments is worse than just copying it.

Erasure coding wins on the two numbers people care about: three failures survived instead of two, at half the storage. What it costs is read complexity and repair traffic. Rebuilding one lost fragment means reading six others, so recovering a failed disk moves several times more data across the network than copying a replica would.

Try it. Knock fragments out one at a time, then switch the placement and cut the power to one rack.

One object, 6 data + 3 parity fragments

Readable, 9 of 9 left

All nine fragments are healthy. Click a fragment to lose it, or cut the power to rack 1.

Placement: one fragment per rack. Rack 1 holds 1 of the 9 fragments, so a power cut there loses 1. Three full copies of the same object would take 3x the space and survive only 2 losses; these nine take 1.5x and survive 3.

The fragments did not change between the two placements. Only where they sat did. That is why placement, the box that picks racks, gets a whole step on the write path.

Back of the envelope#

The eleven nines promise can be checked with a short calculation. An object is lost only if more than 3 of its 9 fragments die before repair puts them back. So the chance of loss depends on how often disks die, and on how long a lost fragment stays lost. Nines here mean the same as in availability and the nines, just counted for losing data rather than being down. If this kind of estimate is new, the back of the envelope chapter walks through the method.

Durability arithmetic

Data fragments6

Parity fragments3

Annual disk failure rate2%

Days to repair a lost fragment1

Fragments per object6 data + 3 parity = 9

Storage overhead9 / 6 = 1.50x

Failures survived3

Chance of loss per object per year4.1e-13

12 nines

at 1.50x storage, if a lost fragment is rebuilt within 1 day

Now drag the repair slider. Durability depends on fixing a lost fragment before enough others fail to cross the line, so the same code that rebuilds in a day is far more durable than one that takes two weeks. That is why placement spreads fragments widely: many machines then share the rebuild work and it finishes faster.

Drag the parity slider and the nines move. Then drag the repair slider and watch them move just as much. The durability claim is not only a property of the coding scheme. It is a property of how fast you repair, which is the same lesson as "how long it takes to recover matters as much as how often you fail" in the availability chapter.

Correlated failure is what actually loses data

All the arithmetic above assumes disks fail independently, one at a time and by chance. They do not. A rack loses power, a batch of drives from one manufacturing run fails the same month, a firmware bug hits every drive of a model at once, or a bad deploy corrupts fragments faster than repair can fix them. Placement is what turns the theoretical number into a real one: spread fragments across racks, power domains and availability zones, so no single shared event can take more than the parity count. You just watched this happen in the widget above.

Step 2: Propose the high level design#

The core entities

ObjectA bucket, a key and a version. Never changed in place: a PUT writes a new version.
FragmentOne of nine pieces of an object, six data and three parity, each on a different rack.
Metadata rowWhere the fragments are and which scheme was used. Writing it is the moment the object exists.

Never breaks: An object exists only once its metadata row is written, and that row is written after its fragments.

The API#

Three calls. A range request asks for only part of an object, and a continuation token is a bookmark the server hands back so the next page of a listing starts where the last one ended.

PUT/{bucket}/{key}

raw bytes, or an initiate multipart upload for large objects

returns 200 { etag, versionId }

Why: Objects are immutable, so a PUT to an existing key writes a new version rather than changing anything. That makes a copy just a copy, makes rollback possible, and removes every concurrency problem an in place update would create.

GET/{bucket}/{key}

returns 200 with bytes, or 206 for a range request

Why: Range requests matter more than they look: a client reading the last kilobyte of a 500GB archive should not transfer 500GB, and with erasure coding this means fetching only the fragments covering that byte range.

GET/{bucket}?prefix=2026/08/&delimiter=/&continuation=...

returns 200 { keys: [...], commonPrefixes: [...], nextContinuation }

Why: Always paged with a continuation token, never an offset. A prefix can match a hundred million keys, and offset paging over a range that is being written to at the same time returns duplicates and misses.

The whole system on one whiteboard#

Figure 1 is the same system as the picture at the top, with every box in its place.

Figure 1. Two planes with different problems. The data plane scales by adding disks and every disk works on its own. The metadata plane holds one small row per object and is the harder half.

Walking Figure 1:

Every request enters through a load balancer and lands on a stateless router: one that keeps nothing about any object between requests. Any router can take any request, so a dead one is simply replaced.
The router asks the metadata service where the object is, or where it is going.
On a write the object is split into nine fragments, six of data and three of parity. On a read, any six of the nine are enough to rebuild it.
Placement decides which racks, rooms and power supplies the nine fragments go to. Nine fragments on one rack is nine copies of the same failure.
Fragments land on storage nodes. Metadata is committed last, which is what makes read after write true: if the metadata row is there, the data is already there.
In the background, the scrubber reads everything on a slow rotation and checks each fragment's checksum, a short fingerprint of its bytes stored alongside it (see the hashing section of hashing vs encryption vs encoding). The repair service rebuilds anything missing or corrupt. Those two dashed arrows are where the eleven nines actually come from. The rest is just storing bytes.

Splitting metadata from data is the big structural decision. The data plane grows by adding disks, and each disk works on its own. The metadata plane holds one small row per object, and at a trillion objects that is the harder half. Listing a prefix has to be fast, and the keys were chosen by users who never thought about how you split your tables.

Your answer

Figure 1 keeps nine fragments and can lose any three. Three full copies costs 3x storage and survives two failures. What does erasure coding give up to beat that?

The data model#

One row per object version. The column that matters most is placement: without it the fragments are nine anonymous pieces scattered across a datacentre.

object_metadatasharded key value store, ordered by key for prefix listing

bucket	varchar(63)	PK
key	varchar(1024)	PK	Kept in order within the bucket, which is what makes prefix listing a range scan rather than a full scan.
version_id	bigint	PK	A PUT writes a new version. Nothing is ever changed in place.
size	bigint
etag	char(32)		A hash of the content. Returned on PUT so a client can check what landed.
placement	list<node_id>		Which nodes hold which fragment. Without this map the bytes cannot be put back together.
ec_scheme	varchar(8)		"6+3" or "repl3". Recorded per object so the scheme can change without rewriting old data.

Sample row

photos | 2026/08/25/img_88.jpg | 41 | 2148291 | 9f3c... | [n12,n88,n41,...] | 6+3

Storing the scheme per object rather than once for the whole system is what lets you migrate: new objects use a better code while old ones stay readable under the one they were written with. A global setting would mean rewriting exabytes to change anything.

Step 3: Design deep dive#

The write path#

The order of the steps in a write decides what a crash leaves behind.

In what order do you write the fragments and the metadata row?

First try
Metadata first, then the fragments
Write the row saying where the nine fragments will live, then write the fragments.
Where it breaks: A crash between the two leaves a row pointing at fragments that were never written. The object looks present in every listing and is gone the moment someone reads it.
Better
Fragments first, wait for all nine, then metadata
Write all nine fragments, wait until every one confirms, then commit the row.
Where it breaks: A crash now only leaves stray fragments, which is safe. But every write waits for the slowest of nine disks, and one slow disk sets the latency of every write it touches.
What you ship
Fragments in parallel, carry on at eight, metadata last
Send all nine at once. When eight have confirmed, six would already rebuild the object, so carry on and let repair fill in the ninth. Then commit the metadata row, and only then answer the client. A sweeper deletes stray fragments that no row points at.

The second rung's problem, waiting on the slowest of many machines, has its own chapter: tail latency at scale. Here is the shipped version as a sequence.

1/6 Large objects arrive as a multipart upload: the client splits the file into parts and uploads them separately, so a failure at 90% costs one part rather than the whole gigabyte.

That ordering is the whole trick, and it is worth saying out loud in an interview: commit the metadata last, so a crash costs you disk space rather than data. Orphan fragments, ones no row points at, are cleaned up by a background sweeper.

The metadata problem#

A trillion objects means a trillion rows, and users choose the keys. One machine cannot hold them, so the rows are split across many machines. That is sharding, and the sharding chapter covers it from the start. The question is how to split.

How do you split a trillion metadata rows across machines?

First try
Hash the key
Run each key through a hash and use the result to pick a machine. Every machine gets an even share, whatever the keys look like.
Where it breaks: Listing a prefix is the one query this API must be good at, and a hash scatters 2026/08/25/a and 2026/08/25/b onto different machines. Every listing has to ask every machine.
Better
Split by key range
Keep keys in sorted order and give each machine a range, like a dictionary split into volumes by letter. A prefix listing reads one contiguous range.
Where it breaks: Users name keys by date. Every object written today shares a prefix, so every write in the system lands on the one machine holding today. Tomorrow the fire moves to the next range.
What you ship
Key ranges that split themselves
Ranges split automatically when they get hot or large. A heavily written prefix becomes many ranges on many machines on its own, and listing stays a range scan.

The sharding chapter compares hash and range splitting in a widget. The classic key that causes trouble looks like 2026-08-25/1430/event.json: the partition holding today is on fire and yesterday's is idle. The third rung is more machinery than hashing, and it keeps the property the API needs. (The same "ordered or hashed" choice runs through design a key value store.)

The follow up you will get

"There are no real directories here, so what does listing a prefix actually do?" It is a range scan over sorted keys, returning keys that share a string prefix. A delimiter parameter makes it look like folders by collapsing everything after the next slash. A bucket with a hundred million objects under one prefix is a hundred million row scan, which is why listing is paged with a continuation token and why nobody should build a filesystem on top of it.

Proving the data is still there#

Bits rot. A disk can return data it believes is correct and is not, sometimes for years before anyone reads it. Storing three copies of a corrupted object is not durability.

How do you know an object nobody has read in three years is still intact?

First try
Trust the disk
If the disk returns bytes without an error, assume they are right.
Where it breaks: Silent corruption is exactly the case where the disk reports no error. The read path turns it into a wrong answer, given confidently.
Better
Check a checksum on every read
Store a checksum with each fragment and verify it whenever the fragment is read. A bad fragment is caught and rebuilt from its peers during a normal GET.
Where it breaks: Most objects are rarely read. Rot can sit unnoticed for years while more fragments of the same object quietly go bad, and by the first read there may be fewer than six good ones.
What you ship
Checksums on read, plus a scrubber and repair
Keep the read check, and add a background scrubber that reads every fragment on a cycle, verifies it, and sends anything bad to repair. Problems are fixed while there are still plenty of good fragments.

The scrub rate is a real capacity decision. Scrubbing everything once a month means reading the whole exabyte fleet monthly in the background, and that reading competes with real traffic for the same disks.

Break it#

Failure scale

one disk

one diskone rackrepair backlogcorrelated failurediversified

Healthy. A drive fails, which happens every day at this scale. Reads still find six of the remaining eight fragments, repair rebuilds the missing one onto a new disk, and no user notices anything. This is normal operation, not an incident.

Read every step as text

one disk · Healthy. A drive fails, which happens every day at this scale. Reads still find six of the remaining eight fragments, repair rebuilds the missing one onto a new disk, and no user notices anything. This is normal operation, not an incident.
one rack · Degraded. A rack loses power. Because placement spread the nine fragments across racks, no object lost more than one or two, so everything is still readable. Repair traffic spikes as thousands of objects rebuild at once, and that traffic now competes with real reads.
repair backlog · Down. Repair is slowed down to protect read latency, so the backlog grows faster than it drains. Every object waiting for repair is an object whose durability is far below the advertised number, because the arithmetic assumed repair finishes quickly. The published figure quietly stops being true.
correlated failure · Down. A firmware bug hits every drive of one model at once. Independence, which every durability calculation assumed, was never real. If those drives are spread evenly across placements, some objects lose four fragments out of nine and are gone. This is how data is actually lost, not through unlucky independent failures.
diversified · Degraded. Placement accounts for shared causes: fragments spread across racks, power domains, availability zones and, where possible, drive models and firmware versions. Repair gets a guaranteed share of capacity rather than being slowed to zero, because an unrepaired fragment is a durability risk and a slow read is only an annoyance.

Trade-offs#

Choice	What you gain	What you pay	Pick it when
Erasure coding over replication	Survives more failures at roughly half the storage cost, which at exabyte scale is the difference between viable and not.	Every read touches many nodes, and rebuilding one fragment reads several others, so repair traffic is much heavier.	Objects above a few hundred kilobytes. Below that, fragment overhead makes replication cheaper.
Commit metadata last	A crash leaves orphan fragments, which waste space, rather than metadata pointing at bytes that were never written.	A sweeper is needed to reclaim orphans, and it must be careful not to delete fragments for a write still in progress.	Any system where a pointer and the thing it points at are written separately.
Range partitioned metadata with automatic splitting	Prefix listing stays a range scan, and hot prefixes split into more partitions by themselves.	Far more machinery than hashing, and each split is a live operation with a cost.	When ordered listing is part of the API. Hashing is simpler and removes the feature.
Continuous background scrubbing	Silent corruption is found and repaired before anyone reads it, which is what makes the durability claim honest.	Reading the entire fleet on a cycle, competing with real traffic for the same disks.	Always. Undetected bit rot means the copies you counted on were never really there.

The level bar#

One moment from this round, answered three ways. Pick the one you think is staff level before the labels appear. (What changes at senior and staff explains the gap.)

Pick the staff level answer

InterviewerRepair traffic after a rack failure is slowing customer reads. Operations wants to throttle repair to almost nothing until evening. What do you say?

Show which is which without guessing

Answer A: Mid level. Every object waiting for repair is below its advertised durability, and a second failure in that window can lose data.
Answer B: Senior. Knows the risk and compromises. Has no rule for how much.
Answer C: Staff. Ranks the two harms correctly, gives a rule for the trade, and orders the repair queue by risk.

Then run the question cold, with a 45 minute timer and no notes, and score the run on the five things this question is really testing.

Score your last run

0 / 10

The durability claim
Where the nines come from
Write order
Metadata
Silent corruption

Pick 0, 1 or 2 on every row. Be the strict interviewer, not the kind friend.

Interview replay#

Interviewer

Three copies or erasure coding?

The core trade, and the numbers should come out immediately.

You

Erasure coding for anything above a few hundred kilobytes. A 6 plus 3 scheme survives three failures at 1.5x storage, where 3x replication survives two at 3x. Better durability for half the cost. What I give up is read simplicity, since a read touches six nodes, and repair cost, because rebuilding one fragment means reading six. For small objects I would use replication, because splitting a 4KB object into nine fragments has more overhead than just copying it.

Both numbers, both costs, and the size threshold. That last part shows this is understood, not recalled.

Interviewer

You claim eleven nines. Where does that number come from?

The question that separates levels. Most candidates quote it without being able to derive it.

You

From the chance that more than the parity count of fragments are lost before repair completes. So it depends on three things: the coding scheme, the disk failure rate, and repair speed, and repair speed is the one people leave out. An identical system that repairs in a week instead of an hour has much worse durability. I would also be honest that the arithmetic assumes independent failures, and correlated failures like a bad firmware batch or a rack losing power are what actually lose data, which is why placement diversity matters more than the coding parameters.

Names repair speed as the hidden variable, then attacks the independence assumption. Both are exactly what a staff interviewer is listening for.

Interviewer

Walk me through a write that crashes halfway.

Testing whether the commit point was thought about.

You

Fragments go to storage nodes first and the metadata row is written last, so the metadata commit is the commit point. If we crash before it, there are fragments on disk that nothing points at, and a sweeper reclaims them. That wastes space for a while. If I committed metadata first, a crash would leave a row pointing at fragments that were never written, which is a lost object that looks present until someone reads it. Wasting space is always the better failure.

States the principle in the last sentence, which is the part you can reuse elsewhere.

Interviewer

A customer names every key with today’s date as the prefix. What happens?

The hot partition, in the form this API actually produces.

You

Every write in that bucket lands on one metadata partition, because metadata is range partitioned by key to keep prefix listing a range scan. So today’s partition is saturated and yesterday’s is idle, and tomorrow the problem moves. Hashing the key would fix it and break listing, which is the one query this API must do well, so instead partitions split automatically when they get hot or large. A heavily written prefix becomes many partitions and ordering is preserved.

Explains why the obvious fix is unacceptable before giving the real one. That structure keeps appearing in strong answers.

Interviewer

How do you know an object you have not read in three years is still intact?

Bit rot, and it is easy to miss entirely.

You

You do not, unless you check. Every fragment has a checksum and a background scrubber reads the fleet on a cycle, verifies them, and rebuilds anything that fails from its peers. Checksums are also verified on every read, so corruption is caught rather than served. The scrub cycle length is a real capacity decision, since scrubbing an exabyte monthly is a lot of background reading competing with real traffic, and I would want measurements before committing to a number.

Opens by admitting the naive answer is wrong, then gives both the background and the read path defence.

Checkpoint#

Checkpoint

1. A 6 + 3 erasure coding scheme compared with 3x replication gives you what?

2. Which factor is most often left out of a durability calculation?

3. Why write fragments before committing the metadata row rather than the other way round?

Worth memorising

Eleven nines of durability out of disks with a 1 to 2 percent yearly failure rate. That gap is the entire design.
6 data + 3 parity survives three failures at 1.5x storage, against 3x for three full copies that survive two.
Metadata is committed after the fragments. That ordering is what makes read after write true.
The nines come from scrubbing and fast repair, not from anything on the write path.

Say this in 60 seconds

The product here is a durability claim, so the design is the machinery that turns unreliable disks into eleven nines. I would erasure code anything above a few hundred kilobytes, six data fragments plus three parity, which survives three failures at 1.5x storage where three replicas survive two at 3x, and fall back to replication for small objects where fragment overhead outweighs the saving. The number nobody mentions in that arithmetic is repair speed, because durability is really the chance of losing more than the parity count before a rebuild finishes, and it assumes independent failures which is never true, so placement spreads fragments across racks, power domains and drive models. Metadata and data are separate systems: metadata is range partitioned by key so prefix listing stays a range scan, with automatic splitting to handle hot prefixes. On write, fragments land first and the metadata row is the commit point, so a crash costs disk space rather than an object. And a background scrubber verifies checksums continuously, because three copies of silently corrupted data is not durability.