Amazon S3 System Design Interview: Building Object Storage with Eleven Nines of Durability
Amazon S3 stores hundreds of trillions of objects and averages well over 100 million requests per second, spread across millions of hard drives (figures from AWS talks in 2023, treat as point-in-time estimates). Its headline promise is 99.999999999% durability, eleven nines, which loosely means that if you store ten million objects you should expect to lose one roughly once every ten thousand years. The hard part is that a single spinning disk gives you only about 120 random IOPS and dies without warning, so the entire design is about turning millions of unreliable disks into one storage system that effectively never loses data.
Object storage exposes a dead simple contract: PUT an object under a key in a bucket, GET it back byte for byte, and never lose it. Underneath, that contract hides two genuinely hard systems. The first is the durability engine that takes each object, splits it into shards using Reed-Solomon erasure coding, and scatters those shards across many disks, racks, and availability zones so that no correlated failure can destroy more shards than the parity can rebuild, all while background scrubbers continuously read, verify checksums, and repair. The second is the metadata index that maps a key to the physical location of its shards, which must stay fast and consistent while holding entries for trillions of keys and absorbing request storms concentrated on a few hot prefixes. On top of those sit a stateless front-end router fleet, a chunked data path with multipart upload for large objects and range reads for partial GETs, and a lifecycle engine that migrates cold data into cheaper tiers. Since 2020 S3 has also offered strong read-after-write consistency, so a read that follows a successful write always sees the latest bytes. The design lesson is that availability comes from stateless replaceable front ends, and durability comes from spreading redundant coded data as widely as possible and never trusting a disk to tell you the truth about itself.
Asked at: Asked at Amazon, Google, Microsoft, and most infrastructure-heavy companies, and it shows up whenever a team needs to store blobs, backups, media, or data lake files at scale. Variants include design Google Cloud Storage, design a photo or video blob store, and design a backup system.
Why this question is asked
Interviewers like object storage because it looks deceptively simple and then opens into the deepest questions in distributed systems. A candidate has to reason about durability math rather than hand wave about replication, has to decide between full replication and erasure coding and defend the storage cost tradeoff, and has to design an index that survives hot partitions at trillions of keys. It also forces a clear separation between the availability of the request path and the durability of the data, which are different properties that people frequently conflate. Strong versus eventual consistency, background repair, and lifecycle tiering give plenty of room to go as deep as the interviewer wants.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Store an object of arbitrary size under a user chosen key inside a bucket, and read it back exactly (byte for byte).
- Support PUT (create or overwrite), GET, DELETE, and HEAD (metadata only) on objects.
- Support multipart upload so a large object can be uploaded in parallel parts and assembled server side.
- Support range reads so a client can fetch a byte range of a large object without downloading the whole thing.
- List objects in a bucket, optionally filtered by key prefix and delimiter, with consistent results.
- Store per object metadata: size, content type, checksum, user tags, storage class, and timestamps.
- Offer multiple storage classes (hot, infrequent access, archive) with lifecycle rules that migrate objects between them automatically.
- Enforce per bucket and per object access control and encryption at rest.
Non-functional requirements
- Durability of eleven nines (99.999999999%) for stored objects, the primary design goal.
- High availability for the request path, on the order of 99.99% or better, independent of individual disk or host failures.
- Strong read-after-write consistency: a read issued after a successful write returns the latest data, and list operations reflect completed writes.
- Elastic throughput that scales with request rate, including sustained high rates concentrated on a single bucket or prefix.
- Low and predictable first byte latency for GETs on hot storage classes.
- Cost efficiency: raw storage overhead should stay well below the 3x of naive triple replication.
- Regional isolation so a failure or overload in one region does not affect others.
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Object count
Hundreds of trillions of objects
AWS stated over 280 trillion objects at FAST '23 and the number keeps climbing. Treat as a point-in-time estimate; the design point is that the key index must address on the order of 10^14 keys.
Request rate
Over 100 million requests per second (aggregate)
Published by AWS in 2023 as an average across the service. This is why the front-end fleet must be fully stateless and horizontally scalable.
Per disk throughput
About 120 random IOPS per drive
A single 7200 rpm hard drive sustains roughly 120 random operations per second, essentially unchanged since 2006. To serve 100M req/s you fan out across millions of drives, so aggregate demand smooths out even though each workload is bursty.
Storage overhead, erasure coding
About 1.4x to 1.5x raw
A Reed-Solomon (k, m) scheme such as (10, 4) stores 14 shards for every 10 of data, so overhead is 14/10 = 1.4x while surviving 4 simultaneous shard losses. Derived from the code parameters; compare to 3x for triple replication.
Per prefix request rate
At least 3,500 writes/s and 5,500 reads/s per prefix
AWS documents this as the floor per partitioned prefix. S3 splits a prefix into more partitions as sustained load grows, so total throughput per bucket has no fixed ceiling. Ten well distributed prefixes can drive roughly 55,000 reads/s.
Durability target math
99.999999999% annual durability
Eleven nines means an expected annual loss probability around 10^-11 per object. Concretely, storing 10 million objects you would expect to lose one about every ten thousand years. Achieved by spreading coded shards across independent failure domains and continuously repairing.
High-level architecture
A request arrives at a regional endpoint and is resolved by DNS to a load balanced fleet of stateless front-end request routers. The router authenticates the caller, checks bucket and object permissions, and parses the operation. For a PUT, it streams the incoming bytes, breaks the object into chunks, and hands each chunk to the storage layer, which erasure codes it into data and parity shards and writes those shards to storage nodes chosen across different disks, racks, and availability zones. Once enough shards are durably written to satisfy the durability policy, the front end records the mapping from the object key to the placement of its shards in the metadata index, which is a separately scaled, partitioned key-value service. Only after the index commit does the PUT return success, which is what makes a following read see the new data. A GET runs the reverse path: the front end looks up the key in the index, learns where the shards live, reads back enough shards from the storage fleet to reconstruct the requested bytes (or just the requested range), verifies checksums, and streams the result. Behind all of this, background fleets never stop working: scrubbers continuously read shards and validate checksums, a repair service rebuilds any shard that is missing or corrupt from the surviving shards, device health monitors drain data off disks that look like they are about to fail, and a lifecycle engine migrates objects between storage classes according to bucket rules. The clean split is that front ends and storage nodes are all replaceable and hold no unique state, while durability lives in the redundancy and the repair loop.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Front-end request routers
A large stateless fleet that terminates TLS, authenticates and authorizes each request, and orchestrates the read or write. Because they hold no durable state, they can be added, removed, or replaced freely, which is where request-path availability comes from. They are the load balancing and admission control point for the whole service.
Metadata index (key to location map)
A partitioned, strongly consistent key-value store that maps bucket plus key to the object's metadata and the physical placement of its shards. It is the source of truth for what exists and where it lives, and it also powers LIST. It is scaled and sharded independently of the bytes because its access pattern (tiny records, huge count, hot keys) is completely different from bulk data.
Storage nodes and the on-disk store
Hosts that own physical disks and persist shards, each shard carrying its own checksum. The lowest layer is a log-structured, append-friendly key-value store on the drive (Amazon's is ShardStore, written in Rust). Storage nodes are dumb and interchangeable by design, so losing one is a routine event, not an incident.
Erasure coding and placement service
The component that splits each chunk into k data shards, computes m parity shards with Reed-Solomon, and decides which failure domains each shard goes to. Placement deliberately spreads shards across disks, racks, and availability zones so that no single correlated failure can take out more than m shards. Reconstruction reads any k of the k+m shards.
Durability engine: scrubbers and repair
Background services that continuously read every shard, verify checksums against silent corruption and bit rot, and monitor device health signals like rising error rates and latency. When a shard is bad or a disk is failing, the repair loop reconstructs the missing shards from survivors and rewrites them to healthy hardware, usually before any human notices.
Data path: chunking, multipart, range reads
The pipeline that turns a byte stream into stored shards and back. Large uploads use multipart upload, so parts go up in parallel and are stitched together on completion, which also makes retries cheap. GETs support range reads so a client fetches only the bytes it needs, and the reader reconstructs just those from the relevant shards.
Lifecycle and tiering engine
A rules driven background system that moves objects between storage classes (standard, infrequent access, archive) based on age or access patterns, and expires objects when policy says so. Migration is asynchronous and transparent: the key stays the same while the physical representation and cost profile change underneath it.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
bucketsbucket_nameowner_account_idregioncreated_atdefault_encryptionversioning_enabledBucket names are globally unique within a partition and act as the top level namespace. A bucket is a flat container: there are no real folders, only keys that happen to contain slashes.
object_indexbucket_nameobject_keyversion_idsize_bytesetag_checksumstorage_classlast_modifiedThe hot metadata table. Partitioned by a hash of bucket plus key prefix so load spreads, and it is the source of truth that GET and LIST read. Version_id supports versioning; a delete may write a tombstone rather than removing the row.
shard_placementobject_keychunk_idshard_idstorage_node_iddisk_idaz_idFor each chunk of an object, records where every data and parity shard physically lives. Reconstruction and repair read this to find k surviving shards. Kept close to object_index so a GET resolves location in one logical lookup.
multipart_uploadsupload_idbucket_nameobject_keypart_numberpart_etaginitiated_atTracks in-progress multipart uploads. Parts land independently and are assembled on CompleteMultipartUpload. Abandoned uploads are garbage collected by a lifecycle rule so orphaned parts do not accumulate cost.
lifecycle_rulesbucket_namerule_idprefix_filtertransition_daystarget_storage_classexpiration_daysPer bucket policy that drives the tiering engine. A background scan matches objects against these rules and schedules transitions or expirations. Rules are declarative; the engine reconciles actual state toward them.
device_healthdisk_idstorage_node_idread_error_ratewrite_latency_p99smart_statuslast_scrub_atFeeds the durability engine. Rising error rates or latency mark a disk as suspect and trigger proactive draining of its shards before it fails outright, which keeps the effective replica count from silently dropping.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
The object storage model versus block and file
Object storage is a flat keyspace of immutable blobs plus metadata, and that shape is what lets it scale. There are no folders, no inodes, and no in-place edits: a bucket holds keys, and a key like photos/2026/cat.jpg is a single opaque string, the slashes are just characters that LIST can use as delimiters to fake a directory view. Objects are effectively immutable, so a PUT to an existing key writes a whole new object and atomically flips the index pointer rather than editing bytes on disk. This is very different from block storage, which offers a raw array of fixed size blocks that a filesystem manages, and from file storage, which offers a POSIX tree with rename, append, and byte level updates. By giving up POSIX semantics and in-place mutation, object storage gets a dead simple, embarrassingly parallel contract: any key can be served from anywhere, writes never have to coordinate with concurrent readers of the old version, and the system can move an object's physical bytes around freely for durability or tiering without the client ever knowing.
How you get eleven nines: erasure coding versus replication
The naive way to survive disk failure is triple replication: keep three full copies and you survive two losses, but you pay 3x storage. Erasure coding does much better. Reed-Solomon splits each chunk into k data shards and computes m parity shards over a finite field, so you store k+m shards and can reconstruct the original from any k of them. A (10, 4) scheme stores 14 shards to protect 10, a 1.4x overhead, yet survives any 4 simultaneous shard losses, which is far stronger protection than 3x replication gives you for less than half the extra storage. Eleven nines comes from combining that math with placement: the k+m shards are scattered across independent failure domains (different disks, racks, power, and availability zones) so that a correlated event like a rack losing power cannot take out more than m shards. The tradeoff is that erasure coding costs CPU to encode and decode and that a read may have to touch several nodes to reconstruct, which is why very small or very hot objects are sometimes also replicated for cheap fast reads while cold bulk data leans on coding for cost.
Continuous integrity checks and self repair
Durability is not a one time property, it decays. Disks suffer bit rot, sectors go bad silently, and firmware occasionally returns the wrong bytes without an error, so a system that only writes redundantly and walks away will slowly lose data. S3 style durability depends on a never ending background loop. Every shard is stored with its own checksum, and scrubber processes continuously read shards across the whole fleet and verify those checksums. When a shard fails verification or a disk goes missing, the repair service treats it as a lost shard, reads k surviving shards for that chunk, reconstructs the missing shard, and writes it back to healthy hardware. On top of reactive repair, device health monitoring watches error rates and latency and proactively drains data off disks that are trending toward failure before they die. The key mental model is that the effective redundancy is always being pulled back up toward the target: failures happen constantly, and the design assumption is that at any instant some shards are already gone, so repair has to be faster than the rate at which independent failures can stack up to m.
The metadata index at trillions of keys and the hot prefix problem
The index that maps keys to shard locations is arguably harder than storing the bytes, because it holds a tiny record for every one of trillions of objects and must stay fast and consistent under skewed load. It is a partitioned key-value store, sharded so that different key ranges live on different index nodes. The classic failure mode is the hot partition: if a workload writes keys that all sort together, for example a timestamp prefix like 2026-07-10-..., every request lands on one partition and you hit its per partition ceiling (documented as at least 3,500 writes/s and 5,500 reads/s per prefix) while the rest of the fleet sits idle. S3 addresses this two ways. It automatically detects sustained load on a prefix and splits that prefix into additional partitions, which can take tens of minutes to settle and returns temporary 503 slowdowns while it rebalances. And it pushes clients toward high entropy key design, spreading load by introducing variability early in the key so hashing distributes requests. The general lesson is that at this scale the index, not the disks, is often the throughput bottleneck, and key naming is a performance decision.
Strong read-after-write consistency
For most of its life S3 was eventually consistent for some operations, which meant that right after a write a read or a LIST could briefly return stale results, and teams built workarounds like external consistency layers (EMRFS consistent view, S3Guard) that tracked recent writes in a separate database. In December 2020 S3 made all operations strongly read-after-write consistent, at no extra cost and with no performance hit. The way to reason about this in a design is that the metadata index is the linearization point. A PUT is not acknowledged to the client until the new object's metadata is durably committed and the index pointer for that key has been atomically updated to reference the new version. Because every GET and LIST resolves through that same authoritative, strongly consistent index rather than through a lazily replicated cache, any read that starts after the PUT returns success is guaranteed to observe the new pointer, and therefore the new data. The design cost is that you cannot serve reads off a stale replica of the index and call it done; the index has to provide strong consistency itself, typically via consensus or a primary that owns each partition, without introducing cross region coordination that would hurt availability.
The data path: PUT, multipart upload, GET, and range reads
On a PUT the front end streams bytes, splits the object into chunks, erasure codes each chunk, and writes the shards out in parallel across storage nodes, acknowledging only once enough shards are durable and the index is updated. For large objects a single stream is fragile and slow, so multipart upload lets the client cut the object into parts, upload them concurrently (and retry any single failed part cheaply), and then call complete, at which point the service assembles the parts into one logical object. This turns a risky multi gigabyte transfer into many independent, retryable units and is how you saturate the network on big uploads. On the read side, a GET resolves the key in the index, reads back k shards per needed chunk, reconstructs, checksums, and streams. Range reads make partial access efficient: because the object is stored as independently addressable chunks and shards, a request for bytes 5MB to 6MB only reads and reconstructs the chunks covering that window instead of the whole object, which is what makes video seeking, columnar analytics, and resumable downloads practical on huge files.
Availability and heat management across millions of disks
Availability of the request path is a different property from durability of the data, and S3 gets it from statelessness plus spreading load. The front-end fleet holds no durable state, so any router can handle any request and unhealthy hosts are pulled out of rotation without data implications. The subtler idea is heat management. A single disk gives only about 120 random IOPS, and any one customer's workload is bursty, so if you pinned a customer's objects to a few disks they would routinely hit the wall. Instead the system scatters different objects across an enormous pool of disks, and because it aggregates millions of independent bursty workloads, the combined demand smooths out and any single burst can borrow capacity from over a million drives at once. This statistical multiplexing is why one customer's spike does not need dedicated headroom sitting idle, and it is only possible because the flat keyspace lets any object live on any disk. Regional isolation completes the picture: each region is an independent failure and capacity domain, so overload or an outage in one does not propagate.
Storage tiers, lifecycle, and background migration
Most stored data is cold, so keeping everything on the fastest, most expensive media wastes money. Object storage offers storage classes that trade retrieval latency and cost: a standard hot tier for frequently accessed data, an infrequent access tier that is cheaper to store but charges more per retrieval, and archive tiers where objects may take minutes to hours to restore but cost a fraction to keep. Lifecycle rules let a bucket owner declare, for example, transition objects to infrequent access after 30 days and to archive after 90, and expire them after a year. A background engine scans objects, matches them against the rules, and performs the transitions asynchronously by re-encoding or relocating the physical bytes onto the target media while leaving the key and metadata pointer unchanged, so the migration is invisible to applications. Archive tiers can use denser, colder hardware and more aggressive erasure coding since they are optimized for cost per stored byte rather than IOPS. The engineering point is that tiering is a physical representation change hidden behind a stable logical name, and the migration itself has to be durable and restartable so a crash mid move never loses or double writes an object.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Erasure coding versus full replication
Erasure coding gives stronger failure tolerance at roughly 1.4x storage instead of 3x, which at exabyte scale is an enormous cost saving. The price is CPU for encode and decode and a read that must gather several shards, so tiny or very hot objects may still be replicated for cheap, fast, single node reads.
Object storage versus file or block storage
Giving up POSIX semantics, in-place edits, and a real directory tree buys a flat, immutable keyspace that is embarrassingly parallel and can scale almost without bound. The cost is that clients cannot append to or partially modify an object in place; they rewrite the whole object, which is fine for blobs and terrible for a database file.
Strong consistency versus eventual consistency
Routing every read through a strongly consistent index removes a whole class of application bugs and the external consistency layers people used to bolt on. It requires the index to provide linearizable updates per key, which is more expensive than serving reads off stale replicas, so the engineering effort moves into making that index both consistent and highly available without cross region coordination.
Separate metadata index versus co-locating metadata with data
Splitting the tiny hot key index from the bulk cold bytes lets each scale on its own axis, since one is IOPS and hot-key bound while the other is capacity and bandwidth bound. The cost is an extra lookup and the operational burden of keeping two large distributed systems consistent with each other on every write.
Prefix based partitioning versus a single global index
Partitioning the index by key prefix spreads load and lets the system split hot prefixes on the fly, which is what allows unlimited aggregate throughput. The downside is the hot partition trap: poorly chosen keys that sort together concentrate on one partition, and rebalancing after a spike is not instant and can surface as temporary 503 slowdowns.
Tiered storage with async migration versus one uniform tier
Tiering cuts cost dramatically for the large fraction of data that is cold, at the price of slower and pricier retrieval for archived objects and a background migration engine that must be durable and restartable. A single tier is simpler and always fast but wastes money keeping rarely read data on premium media.
How Amazon S3 actually does it
Amazon has been unusually open about how S3 is built. Andy Warfield's 2023 USENIX FAST keynote and Werner Vogels's writeup describe S3 as hundreds of microservices with a stateless front-end fleet, a namespace or index service, a storage fleet, and background durability operations, running across millions of hard drives with each drive still limited to about 120 IOPS. Durability rests on Reed-Solomon erasure coding with shards placed across independent failure domains plus continuous scrubbing and repair, and the lowest storage layer, ShardStore, was rewritten in Rust as an append-only log-structured store and validated with lightweight formal methods. The move to strong read-after-write consistency in December 2020 was implemented so that all GET, PUT, and LIST operations are strongly consistent at no extra cost. AWS documents the per prefix request-rate floors and the automatic prefix partitioning behavior in its performance guidance, which is the public window into how the index handles hot spots.
Sources
- Werner Vogels: Building and operating a pretty big storage system called S3
- USENIX FAST '23 keynote, Andy Warfield: Building and Operating a Pretty Big Storage System
- AWS News: Amazon S3 Update, Strong Read-After-Write Consistency
- AWS docs: Best practices design patterns, optimizing Amazon S3 performance
- AWS docs: Performance design patterns for Amazon S3 (prefix scaling)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Amazon S3.