Design object storage
Amazon publishes a durability figure for S3 of eleven nines. That is a claim that if you store ten million objects, you should expect to lose one of them roughly once every ten thousand years.
Disks do not have eleven nines. A good drive fails something like once every few years. The entire design of this system is the machinery that turns unreliable hardware into that number, and the interesting question is not how to store bytes, it is how to make a claim like that and defend it.
Three full copies of your data tolerates two disk failures and costs 3x storage. Can you tolerate more failures for less than 3x? What would you give up?
What you are building
- PUT, GET and DELETE of immutable objects by key. Objects are replaced wholesale, never edited in place.
- Objects from a kilobyte to several terabytes. That range is five orders of magnitude and it forces two different code paths.
- Extreme durability, with the arithmetic to back the claim. This is the product. Availability is secondary to not losing data.
- Flat key namespace with prefix listing. There are no directories, and pretending otherwise causes the classic hot partition.
- Trillions of objects, exabytes total.
- Durability target of eleven nines, defensible with arithmetic.
- Read after write consistency for new objects.
- A whole datacentre can fail without data loss.
- A POSIX filesystem interface. Random writes into the middle of an object change everything, which is why this is object storage and not a filesystem.
- Access control and encryption details, which are real and large but structurally separable.
- Lifecycle tiering to cold archival storage, which is a scheduling problem on top of this.
Replication or erasure coding
This is the decision the whole system turns on, and it has a satisfying answer.
Erasure coding wins on the two numbers people care about, tolerating three failures instead of two at half the storage cost. What it costs is read complexity and repair traffic: rebuilding one lost fragment means reading six others, so recovering a failed disk moves several times more data across the network than replication would.
The number that matters most here is not in the sliders: it is repair speed. Durability depends on fixing a lost fragment before enough others fail to pass the threshold, so a system that repairs in an hour is far more durable than an identical one that repairs in a week. This is why placement spreads fragments widely, since a wide spread means many machines contribute to a rebuild and it finishes faster.
Drag the parity slider and watch the nines move. Then note the sentence under it: the durability claim is not really a property of the coding scheme, it is a property of how fast you repair.
All the arithmetic above assumes disks fail independently. They do not. A rack loses power, a batch of drives from one manufacturing run fails the same month, a firmware bug hits every drive of a model at once, or a bad deploy corrupts fragments faster than repair can fix them. Placement is what turns the theoretical number into a real one: spread fragments across racks, power domains and availability zones, so no single correlated event can take more than the parity count.
The design
Splitting metadata from data is the structural decision. The data plane scales by adding disks and is embarrassingly parallel. The metadata plane holds one small row per object, and at a trillion objects that is a database problem harder than the storage problem, because listing a prefix has to be fast and keys are chosen by users who did not think about your partitioning.
| bucket | varchar(63) | PK | |
| key | varchar(1024) | PK | Ordered within the bucket, which is what makes prefix listing a range scan rather than a full scan. |
| version_id | bigint | PK | A PUT writes a new version. Nothing is ever modified in place. |
| size | bigint | ||
| etag | char(32) | Content hash. Returned on PUT so a client can verify what landed. | |
| placement | list<node_id> | Which nodes hold which fragment. This is the map without which the bytes are unrecoverable. | |
| ec_scheme | varchar(8) | "6+3" or "repl3". Recorded per object so the scheme can change without rewriting old data. |
Deep dive one: the write path
That ordering is the whole trick and it is worth saying explicitly in an interview: commit the metadata last, so a crash costs you disk space rather than data. Orphan fragments are cleaned up by a background sweeper that finds fragments with no metadata row pointing at them.
Deep dive two: the metadata problem
A trillion objects means a trillion rows, and users choose the keys.
The classic failure is a key pattern like 2026-08-25/1430/event.json. Every object
written today shares a prefix, so if metadata is range partitioned by key, every write in
the system lands on one partition. The partition holding today is on fire and yesterday’s
is idle, and tomorrow the fire moves one partition to the right.
Hashing the key fixes the hot spot and breaks prefix listing, which is the one query this API has to be good at. The workable answer is range partitioning with automatic splitting: partitions split when they get hot or large, so a heavily written prefix becomes many partitions on its own, and listing remains a range scan. It is more machinery than hashing and it keeps the property the API needs.
“There are no real directories here, so what does listing a prefix actually do?” It is a range scan over an ordered key space, returning keys sharing a string prefix, with a delimiter parameter that makes it look like folders by collapsing everything after the next slash. A bucket with a hundred million objects under one prefix is a hundred million row scan, which is why listing is paginated with a continuation token and why nobody should build a filesystem on top of it.
Deep dive three: proving the data is still there
Bits rot. A disk returns data it believes is correct and is not, sometimes for years before anyone reads it. Storing three copies of a corrupted object is not durability.
Every fragment carries a checksum written with it. A background scrubber continuously reads fragments, verifies checksums, and rebuilds anything that fails from its peers. The scrub rate is a real capacity decision: scrubbing everything once a month means reading your entire exabyte fleet monthly in the background, and that traffic is competing with real reads.
Checksums are verified on every read too, so a corrupted fragment is caught and repaired during a normal GET rather than being served. This is the part people forget: a read path that trusts the disk turns silent corruption into a wrong answer given confidently.
Break it
The API
raw bytes, or an initiate multipart upload for large objects
Trade-offs
| Choice | What you gain | What you pay | Pick it when |
|---|---|---|---|
| Erasure coding over replication | Tolerates more failures at roughly half the storage cost, which at exabyte scale is the difference between viable and not. | Every read touches many nodes, and rebuilding one fragment reads several others, so repair traffic is much heavier. | Objects above a few hundred kilobytes. Below that, fragment overhead makes replication cheaper. |
| Commit metadata last | A crash leaves orphan fragments, which waste space, rather than metadata pointing at bytes that were never written. | A sweeper is required to reclaim orphans, and it must be careful not to delete fragments for an in-flight write. | Any system where a pointer and the thing it points at are written separately. |
| Range partitioned metadata with auto splitting | Prefix listing stays a range scan, and hot prefixes split into more partitions automatically. | Far more machinery than hashing, and splits themselves are a live operation with a cost. | When ordered listing is part of the API. Hashing is simpler and removes the feature. |
| Continuous background scrubbing | Silent corruption is found and repaired before anyone reads it, which is what makes the durability claim honest. | Reading the entire fleet on a cycle, competing with real traffic for the same disks. | Always. Undetected bit rot means the copies you counted on were never really there. |
Interview replay
Checkpoint
1. A 6 + 3 erasure coding scheme compared with 3x replication gives you what?
2. Which factor is most often left out of a durability calculation?
3. Why write fragments before committing the metadata row rather than the other way round?
The product here is a durability claim, so the design is the machinery that turns unreliable disks into eleven nines. I would erasure code anything above a few hundred kilobytes, six data fragments plus three parity, which tolerates three failures at 1.5x storage where three replicas tolerate two at 3x, and fall back to replication for small objects where fragment overhead outweighs the saving. The number nobody mentions in that arithmetic is repair speed, because durability is really the chance of losing more than the parity count before a rebuild finishes, and it assumes independent failures which is never true, so placement spreads fragments across racks, power domains and drive models. Metadata and data are separate systems: metadata is range partitioned by key so prefix listing stays a range scan, with automatic splitting to handle hot prefixes. On write, fragments land first and the metadata row is the commit point, so a crash costs disk space rather than an object. And a background scrubber verifies checksums continuously, because three copies of silently corrupted data is not durability.
