Amazon S3 Object Storage System Design

Amazon Simple Storage Service (S3) is the canonical example of an internet-scale object store — a key-value system where the value is a blob (typically megabytes to terabytes), the key is a UTF-8 string up to 1024 bytes, and the access pattern is HTTP-based PUT / GET / DELETE / LIST against a bucket-namespaced URL space. S3 launched in March 2006 (Vogels’ launch post), is designed to provide eleven nines of durability (99.999999999% over a given year — AWS frames this as a design objective, illustrated as roughly one object lost per 10,000 years if you store ten million objects) by redundantly storing each object “across a minimum of three Availability Zones” so it can “sustain data in the event of the loss of an entire Amazon S3 Availability Zone” (AWS DataDurability docs), was famously eventually consistent for object listing and overwrite-PUT semantics for over a decade, and converted to strong read-after-write consistency for all operations on December 1, 2020 (AWS announcement). S3’s scale is publicly reported at AWS Pi Day each March: 100 trillion objects in 2021, 280 trillion in 2023, 350 trillion in 2024, and over 400 trillion objects processing ~150 million requests/second as of Pi Day 2025 (AWS Pi Day 2025), spanning exabytes of data. Its near-clones — Google Cloud Storage, Azure Blob Storage, and the open-source self-hostable MinIO and Ceph RGW — share the same architectural shape: front-end web tier → metadata index → erasure-coded bytestore → background repair. The story is most legible against its predecessor, the file-system Google File System — both solve “store many big blobs cheaply on commodity hardware” but expose radically different APIs and consistency contracts.

1. Functional and Non-Functional Requirements

Functional.

• Object storage keyed by (bucket, key). Value is an opaque byte blob, typically 0 bytes to 5 TiB (S3’s per-object cap).
• HTTP / HTTPS API. Bucket lives at https://<bucket>.s3.<region>.amazonaws.com/<key>. Verbs: PUT, GET, DELETE, HEAD, LIST (with prefix and delimiter filters), COPY.
• Bucket-level configuration. Versioning (every PUT becomes a new version, deletes are tombstones), object lifecycle (transition to colder tier, expire, abort multipart uploads), CORS (Cross-Origin Resource Sharing), encryption settings (server-side encryption with S3-managed keys / KMS-managed keys / customer-provided keys), access logging.
• Access control. IAM (Identity and Access Management) policies, bucket policies, object ACLs (Access Control Lists), and pre-signed URLs that grant time-bounded access without sharing credentials.
• Multipart Upload (MPU). A large object is split into parts (5 MB to 5 GB each, up to 10,000 parts), each PUT independently and possibly in parallel, then a final CompleteMultipartUpload assembles them.
• Storage classes / tiers. Standard, Intelligent-Tiering, Infrequent Access (Standard-IA), One-Zone-IA, Glacier Instant Retrieval (millisecond access), Glacier Flexible Retrieval (minutes to hours), Glacier Deep Archive (within 12 hours, lowest cost). Lifecycle policies migrate objects between tiers. See §7.5 for the retrieval-time ladder.
• Event notifications. PUT or DELETE events publish to SNS / SQS / Lambda; commonly fed into a distributed log downstream.
• Cross-Region Replication (CRR). Async copy of every PUT into a bucket in a different AWS region.
• Strong consistency. Since December 2020: read-after-write for new objects, read-after-update for overwrites, list consistency across all operations.

Non-functional.

• Eleven nines of durability (99.999999999%) over a given year — a design objective, not an SLA. AWS states the regional storage classes “redundantly store objects on multiple devices across a minimum of three Availability Zones” (AZs — physically separate datacenters within a region, “many kilometers” apart but within 100 km, each with independent power and network), “designed to handle concurrent device failures by quickly detecting and repairing any lost redundancy,” and “regularly verify the integrity of your data using checksums” (AWS DataDurability docs). AWS does not publish whether intra-AZ redundancy is replication or erasure coding; the math in §7.2 shows why a high-parity erasure code plus fast repair is the only way to reach eleven nines.
• Four nines of availability (99.99%) — Standard tier. Lower tiers (IA, One-Zone-IA) trade availability and AZ count for price.
• Exabyte scale. Single-bucket capacity is unbounded; AWS reports exabyte-scale single buckets in customer testimonials.
• High-throughput per-prefix. Originally limited (the famous “use random prefix” workaround), now “at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix,” with “no limits to the number of prefixes in a bucket” — so 10 prefixes parallelize reads to ~55,000 req/s, scaling linearly (AWS optimizing-performance docs).
• Eventually consistent → strongly consistent (December 1, 2020). A landmark architectural change discussed below.
• Global namespace. Bucket names are unique across all of AWS — a deliberate choice that simplifies URL routing at the cost of a global serialization point at bucket creation.

2. Capacity Estimation

Take a simplified single-region S3 deployment serving ten exabytes (10 × 10^18 B) of customer data:

• Raw capacity behind 11-nines durability with erasure coding. Suppose each AZ uses a Reed-Solomon (k=10, m=4) code (10 data shards + 4 parity = 14 shards total; survives 4 simultaneous shard losses). The space overhead is 14/10 = 1.4×, far better than 3-replica’s 3×. Across 3 AZs with the encoded copy in each, total raw is 3 × 1.4 = 4.2× the user data, or 42 EB raw. (S3 in practice does not triplicate the EC stripes to all 3 AZs — see §7 — this is a pedagogical figure to show order of magnitude.)
• Storage nodes. With 16-TB drives and ~30 drives per node = ~480 TB raw per node. 42 EB / 480 TB ≈ 88 million drive-equivalents, or ~3 million storage nodes. (Real S3 numbers are not public; this is a thought experiment to demonstrate the scale.)
• Object count. Average object size in real workloads varies wildly — large media files average MBs, small thumbnails KB. For round numbers, 10 EB / 100 KB avg = 10^14 objects = 100 trillion (the right order of magnitude — AWS’s publicly reported count was 100 trillion in 2021 and surpassed 400 trillion by Pi Day 2025).
• Metadata. Each object has a metadata record: bucket, key, version, size, ETag (MD5 of content), storage class, encryption key reference, ACL, last-modified. Estimate 1 KB per object. 10^14 × 1 KB = 100 PB just for metadata. This must live in a distributed key-value store (S3’s index layer, internal name not public; rumored to use a derivative of Dynamo / DynamoDB-like sharded LSM-backed store).

Throughput math.

• Each storage node has, say, a 25 Gbps NIC = 3.1 GB/s. 3M nodes × 3.1 GB/s = 9.3 PB/s aggregate raw bandwidth at the disk-server layer. Network fabric and the front-end tier are the practical limits.
• Frontend tier: AWS reports tens of millions of requests per second to S3 globally.
• A single client streaming a single object from one storage node tops out at NIC line rate, typically ~100 MB/s for an EC2 instance with a moderate NIC. To go faster a client uses range GETs in parallel (read different byte ranges from different replicas).

The capacity story for S3 is two-tiered: a cheap, exabyte-scale, erasure-coded bytestore at the bottom; a hot, in-RAM-resident, sharded metadata index on top. The index is the engineering crown jewel.

3. Application Programming Interface

3.1 Core REST API

PUT /{key}            HTTP/1.1
Host: {bucket}.s3.{region}.amazonaws.com
Authorization: AWS4-HMAC-SHA256 ...
Content-MD5: <base64 md5 of body>
Content-Length: <n>
x-amz-storage-class: STANDARD | STANDARD_IA | GLACIER | ...
x-amz-server-side-encryption: AES256 | aws:kms

<body bytes>
-----
HTTP/1.1 200 OK
ETag: "<md5 hex>"
x-amz-version-id: <opaque>

Simplest possible API: a path-mapped HTTP method. The Authorization header is AWS Signature Version 4 — the request body and headers are HMAC-SHA256-signed with a key derived from the user’s IAM credentials.

GET mirrors PUT. DELETE removes the object (or, if versioning is on, places a delete-marker). LIST returns up to 1000 keys at a time with a NextContinuationToken for pagination; supports prefix= and delimiter= for hierarchical listing (this is how the “folder” abstraction in S3 console UIs is implemented — purely client-side parsing of /-delimited keys).

3.2 Multipart Upload Sequence

1) POST /<key>?uploads          → returns UploadId
2) PUT /<key>?partNumber=1&uploadId=<U>   (body = part 1 bytes)  → returns ETag_1
   PUT /<key>?partNumber=2&uploadId=<U>   (body = part 2 bytes)  → returns ETag_2
   ... in parallel ...
3) POST /<key>?uploadId=<U>     (body = list of (partNumber, ETag)) → assembles object

If step 3 is never invoked, partial state lingers; lifecycle rules with AbortIncompleteMultipartUpload reclaim it. Pre-signed URLs allow each part PUT to be performed by an end-user browser without exposing AWS credentials.

3.3 Pre-Signed URLs

A pre-signed URL is a regular S3 URL with the AWS Signature Version 4 baked into query parameters:

https://bucket.s3.region.amazonaws.com/key
  ?X-Amz-Algorithm=AWS4-HMAC-SHA256
  &X-Amz-Credential=...
  &X-Amz-Date=20260508T120000Z
  &X-Amz-Expires=900
  &X-Amz-Signature=<hex>

Anyone with the URL can perform the corresponding GET / PUT until X-Amz-Expires seconds elapse. The pattern of “your backend issues a pre-signed URL, the browser uploads directly to S3” is one of the canonical AWS architecture moves — it offloads the data path entirely from your backend.

4. Data Model

S3’s data model is flat by design. Objects live directly inside a bucket; there are no real directories. The two-level hierarchy is (bucket, key) only.

Bucket. A globally-unique name (across all of AWS) that anchors a region, owner, and bucket-level configuration (versioning, replication, lifecycle, default encryption, public access block).

Object. A unit identified by (bucket, key, [versionId]). Stores:

• The byte payload, possibly encrypted at rest, possibly erasure-coded across many storage nodes.
• A small set of system metadata (size, ETag, last-modified, storage class, encryption metadata).
• A small set of user-defined metadata (x-amz-meta-* headers).
• An ACL or inherited bucket policy.

Version. When versioning is enabled, every PUT to an existing key creates a new version (an opaque version identifier) rather than overwriting. Versions are listed in newest-first order. A DELETE places a delete marker — itself a version with no payload — that hides earlier versions on subsequent GETs but does not destroy them. Permanent deletion uses DELETE ?versionId=....

The key as a fake hierarchy. S3 keys often look like paths (photos/2023/jan/a.jpg). The / is not meaningful to S3 itself — it’s a string. The LIST operation can use delimiter=/ to interpret / as a folder boundary on the read side, but objects are stored in a flat keyspace.

5. High-Level Architecture

flowchart TB
    subgraph Edge["Edge / Frontend"]
        DNS[Route 53 DNS<br/>bucket.s3.region.amazonaws.com]
        FRONT[S3 Frontend Tier<br/>auth, signature, routing<br/>thousands of stateless web servers]
    end

    subgraph Index["Metadata Index Layer"]
        IDX[(Distributed KV Index<br/>sharded by bucket+key prefix<br/>LSM-backed, replicated)]
    end

    subgraph Bytestore["Bytestore / Storage Plane"]
        SP[Storage Placement Service]
        EC[Erasure Coding Layer<br/>Reed-Solomon e.g. 10+4]
        SN1[(Storage Node 1)]
        SN2[(Storage Node 2)]
        SNn[(Storage Node N)]
    end

    subgraph Background["Background Workers"]
        AE[Anti-Entropy / Repair]
        GC[Garbage Collector]
        LC[Lifecycle Engine]
        CRR[Cross-Region Replication]
    end

    Client((Client)) --> DNS --> FRONT
    FRONT -->|metadata lookup/write| IDX
    FRONT -->|read or write bytes| EC
    EC <--> SN1
    EC <--> SN2
    EC <--> SNn
    EC <--> SP
    AE --> SN1
    AE --> SN2
    GC --> IDX
    LC --> IDX
    LC --> EC
    CRR --> IDX

What this diagram shows. A request arrives at one of thousands of stateless frontend web servers behind virtual-hosted-style DNS. The frontend authenticates the request (validates the AWS Signature Version 4 against the principal’s IAM credentials), parses the bucket and key, and looks up the metadata index to find (a) the object’s placement — which storage nodes hold which erasure-coded shards — and (b) authorization (bucket policies, ACLs). For a GET, the frontend then reads enough shards from the storage plane to reconstruct the object (k of n shards in Reed-Solomon). For a PUT, the frontend writes shards and updates the index. Background workers handle (a) anti-entropy — periodic Merkle-tree comparison between replicas to detect bit rot or missed writes, (b) garbage collection — reclaim shards orphaned by overwrite or delete, (c) lifecycle — transition objects to cold tiers and expire old versions, and (d) cross-region replication — async copy of every PUT to a bucket in another region. The architectural separation between metadata index and bytestore is the central design move: metadata is small, hot, and demands strong consistency; bytes are huge, cold-ish, and tolerate eventual consistency in storage placement.

6. Request Flow

6.1 PUT (Single-Part)

sequenceDiagram
    actor Client
    participant FE as S3 Frontend
    participant IDX as Metadata Index
    participant SP as Storage Placement
    participant SN as Storage Nodes (multiple)

    Client->>FE: PUT /key (bytes B)
    FE->>FE: validate Signature V4, parse bucket
    FE->>IDX: locate bucket, get policy, check ACL
    IDX-->>FE: bucket metadata
    FE->>SP: request placement for object (size, class)
    SP-->>FE: placement plan (storage nodes for each shard)
    FE->>FE: erasure-encode B into k+m shards
    par
      FE->>SN: write shard 1
      FE->>SN: write shard 2
      FE->>SN: write shard ... k+m
    end
    SN-->>FE: shard acks (need >= k+m for durability)
    FE->>IDX: commit metadata: key -> version, shard map, ETag
    IDX-->>FE: commit ack (strongly consistent since 12/2020)
    FE-->>Client: 200 OK, ETag

The committed metadata is what makes this a strongly-consistent write: until the index commit succeeds, a subsequent GET of the same key will not see the new version. Once it commits, all readers (in the same region) see the new bytes — this is the guarantee added in December 2020.

6.2 GET

The mirror image: frontend looks up metadata in the index, fetches at least k shards from the storage nodes (in parallel for low tail latency), reconstructs the object, streams it to the client. If a shard read fails, the EC reconstruction can pull a different shard — a key reason erasure coding is preferred over plain replication: with 3-replica you can lose at most 2; with (10, 4) you can lose 4 out of 14, so transient slow nodes are easily routed around without the durability hit.

6.3 LIST

LIST is interesting because it is not a point query. It must enumerate keys with a given prefix in lexicographic order. The metadata index is sharded by key (or by some hash of bucket+key); LIST may need to query multiple shards. Result pages are bounded (1000 keys); the NextContinuationToken lets the client resume.

7. Deep Dive — Selected Topics

7.1 Erasure Coding vs Plain Replication

S3 publicly states 11-nines durability but does not publish the exact erasure coding scheme. The general principle, from Reed-Solomon literature (Reed & Solomon 1960, Plank survey):

A Reed-Solomon (k, m) code splits the object into k equal-sized data shards and computes m additional parity shards by treating data as polynomial coefficients over a finite field GF(2^8) and evaluating that polynomial at m extra points. Any k of the k+m shards are sufficient to reconstruct the object — in linear-algebra terms, the encoding matrix is constructed so that any k-row submatrix is invertible.

For (k=10, m=4):

• Storage overhead: 14/10 = 1.4× (vs 3× for triple-replication).
• Tolerated simultaneous shard loss: 4 of 14.
• Reconstruction cost: read k = 10 shards from any healthy nodes, do a finite-field matrix multiply.

The durability vs cost trade is dramatic. A 1-EB customer holding their data in 3-replica needs 3 EB raw; in (10, 4) EC they need 1.4 EB raw — a ~2.1× cost saving on the dominant cost center of the entire service.

The penalty: encode/decode CPU, read amplification on small reads (must touch k nodes for any read, vs 1 for a replica), and slow rebuild (rebuilding a lost shard requires reading k others — for k = 10, this is 10× the data movement of a replica rebuild). Production EC schemes use local reconstruction codes (LRC) to mitigate the rebuild cost — Microsoft’s Azure Storage paper (Calder et al. SOSP 2011) describes their LRC variant.

7.2 Durability Math — Where Does 11 Nines Come From?

Assume independent failure of disks, each with annual failure probability p = 4% (Schroeder & Gibson FAST 2007 measured 2–4% in production fleets, much higher than vendor MTBF specs would predict).

For an object stored as n independent replicas, the probability that all replicas fail in a given year is roughly p^n. For n = 3, p^3 = 6.4 × 10^-5 — that’s only about 5 nines, not eleven. So plain 3-replication is not enough.

The eleven-nines claim depends on:

1. Repair faster than failure. If the system detects a lost replica and rebuilds within hours, the window during which the object is under-replicated is small. The probability that a second replica fails during the rebuild window is what matters, not the annual probability.
2. Erasure coding with high m. With (k=10, m=4), losing 4 shards before rebuild completes is much rarer than losing 2 of 3 replicas.
3. Cross-AZ placement. Failure modes that take down a whole AZ (power, fire, bad firmware push) are not independent within the AZ. Spreading shards across multiple AZs converts AZ-correlated failure into independent shard losses at the cluster level.

The combinatorial calculation, given hourly repair, AZ-level failure independence, and (k, m) = (10, 4): the probability of object loss in a year is on the order of 10^-12 to 10^-11. AWS publishes 11 nines as the conservative figure.

Uncertain

Verify: the exact intra-AZ redundancy scheme (replication vs erasure coding), the k/m parameters, and repair-time targets. Reason: AWS officially confirms only the outcomes — “minimum of three Availability Zones,” checksummed integrity verification, “quickly detecting and repairing any lost redundancy,” and the 11-nines design objective (AWS DataDurability docs) — but never the encoding internals. The (10,4) figures used above are illustrative industry-standard reasoning that reproduces 11 nines, not AWS-published numbers. To resolve: an AWS durability methodology paper (none public as of 2026-05). uncertain

7.3 The Eventually-Consistent → Strongly-Consistent Conversion (December 2020)

Through 2020, S3’s documented model was read-after-write consistency for new objects only, plus eventual consistency for overwrites and deletes, plus eventual consistency for LIST. Concretely:

• PUT a new key, then GET — would always return the new bytes.
• PUT an existing key (overwrite), then GET — might return the old bytes for some seconds.
• DELETE then GET — might return the old bytes for some seconds.
• LIST might or might not include very recent PUTs.

This was acceptable for many workloads but a sharp edge for others. Customers wired up workarounds: in-app caching with versioned keys, write-and-poll-with-If-Modified-Since loops, read-from-DynamoDB-then-S3 patterns.

On December 1, 2020, AWS announced strong consistency for all S3 operations — “all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata,” applied to “all existing and new S3 objects,” in all regions, “at no extra charge” and with “no impact on performance” (AWS announcement). Vogels’ follow-up post describes the actual mechanism, and it is not a write-path serialization point — it is a cache-coherence protocol borrowed in spirit from CPU caches. S3’s metadata subsystem fronts a durable persistence tier with a cache for hot object metadata; the danger is a GET reading a stale cached entry after a recent write. Rather than abandon the cache (which would tank latency), AWS added a new witness component that “acts as a witness to writes, notified every time an object changes” and tracks “minimal, in-memory state without disk persistence.” On the read path the witness functions as a read barrier: the cache consults the witness to learn whether its cached view is stale; if fresh, the cached value is served, and if stale it is invalidated and re-read from the persistence tier. New replication logic in the persistence tier orders per-object operations so the witness’s verdict is authoritative. Because witnesses hold only tiny in-memory state, they “achieve extremely high request processing rates with very low latency,” which is how AWS delivered the upgrade with no measurable read or write penalty.

The change was deployed transparently — no API changes, no opt-in. Customers got the new contract for free, with no performance or availability tradeoff (a notable engineering feat: retrofitting strong consistency onto a live exabyte-scale eventually-consistent system).

Uncertain

Verify: the wire protocol and durability/quorum details of the witness↔cache↔persistence-tier interaction. Reason: Vogels’ blog gives the concepts (witness as read barrier, cache-coherence protocol, persistence-tier replication ordering) but not the protocol, replication factor, or failure-mode behavior. To resolve: a deeper AWS engineering disclosure or conference talk. The named mechanism above is officially sourced; anything more specific is inference. uncertain

7.4 Partitioning by Key Prefix — and the “Random Prefix” History

S3’s metadata index is partitioned by key. Historically this meant the first few characters of the key determined the shard. A workload that wrote sequential keys like 2023/01/01/..., 2023/01/02/... would hammer one shard and saturate it. AWS’s pre-2018 guidance was: prefix your keys with a random hash to spread the load, e.g., <random-hex-4>/2023/01/01/key.

In July 2018, AWS announced an updated performance model where the prefix-based limits were replaced with per-prefix scaling (3,500 PUT/sec, 5,500 GET/sec per prefix) and the index now automatically resharded hot prefixes. The “random prefix” advice was deprecated. The historical artifact remains in many bucket schemas.

Internally, this represents the index moving from a fixed prefix-range partitioning scheme to adaptive resharding (similar to what DynamoDB does — splitting hot partitions transparently). A Bloom Filter is plausibly used to skip prefix probes that are guaranteed to be empty during LIST operations across many shards, though AWS does not document this.

7.5 Glacier and Storage Tier Migration

The Glacier family comprises three storage classes optimized for cold data — data accessed less than monthly — with a deliberate latency-for-cost ladder (AWS storage classes): S3 Glacier Instant Retrieval serves data “in milliseconds with the same performance as S3 Standard” (for rarely-accessed-but-must-be-instant data); S3 Glacier Flexible Retrieval (formerly just “Glacier”) offers “configurable retrieval times, from minutes to hours, with free bulk retrievals” via an explicit RestoreObject call; and S3 Glacier Deep Archive is the coldest, with “retrieval time within 12 hours.” All three still carry the 11-nines durability objective and span a minimum of three AZs. The on-disk medium for the deep tiers has historically been described as “tape-like” (whether literal tape or just powered-down disks is internal).

The economics: Glacier Deep Archive is ~$1/TB/monthvsStandardat$23/TB/month — a 23× cost reduction. The latency tax is the trade. Lifecycle rules automate migration: “transition objects older than 30 days to IA, older than 90 days to Glacier, older than 365 days to Deep Archive.”

7.6 Event Notifications and the Stream-Data-Out Pattern

S3 publishes per-object events (PUT, DELETE, lifecycle transition) to:

• SNS (Simple Notification Service — fanout to many subscribers)
• SQS (Simple Queue Service — durable point-to-point)
• Lambda (serverless function trigger)

This is the S3-as-a-data-source pattern: an upstream system PUTs files, a downstream pipeline reacts on the event. A Kafka-style log can be the consumer if the SQS queue is drained by a connector. This integration is one of the most-used patterns in cloud architecture and explains why so many ETL pipelines start with “drop the file in S3.”

8. Scaling Considerations

S3 scales by:

• Stateless frontend. Trivially horizontal — add more web servers, route traffic via Route 53 weighted DNS or anycast.
• Index resharding. The metadata index is sharded by key; hot shards are split automatically. The index itself is a distributed KV store (cf. Distributed Key Value Store System Design) running underneath S3.
• Erasure-coded storage spread across millions of nodes. Each object’s shards are placed on a small subset; aggregate cluster bandwidth scales with node count.
• Per-prefix request rate. With auto-resharding, request rate per logical prefix scales linearly by adding prefixes (which is mostly a key-design choice for the customer).
• Cross-region replication. Each region is independent; replication crosses on a per-bucket basis. A global CDN (CloudFront) caches popular objects at the edge.
• Multi-AZ resilience. Each region has multiple AZs (typically 3). Object shards spread across AZs survive single-AZ failure (power, network, fire).

9. Real-World Examples

• Amazon S3 (launch blog 2006, strong consistency announcement 2020) — the original. AWS’s publicly reported scale grew from 100 trillion objects (2021) to over 400 trillion objects processing ~150 million requests/second by Pi Day 2025.

• Google Cloud Storage — same architectural shape; built atop Colossus (the GFS successor) plus a Spanner-backed metadata layer (which gives it strong consistency from day one — Google’s storage was strong-consistent before S3 caught up in 2020).

• Azure Blob Storage (Calder et al. SOSP 2011) — the third hyperscaler. Notable for being designed strong-consistent from the start and for its detailed publication of the architecture: stream layer (replicated append-only log), partition layer (object index built atop streams), front-end layer. Reading the Azure Storage paper is one of the best ways to understand what an S3-class system looks like inside; AWS does not publish a comparable paper.

• MinIO — open-source S3-compatible object store, deployable on commodity hardware, used inside many private clouds. Its erasure coding is configurable; default is (k=N/2, m=N/2) for an N-node cluster.

• Ceph RGW (RADOS Gateway) — open-source object store atop Ceph’s RADOS distributed object store; provides S3 and Swift APIs.

• The 2017 S3 outage (February 28, 2017). AWS post-mortem: an authorized S3 team member ran a command intended to remove a small number of servers for an established billing-subsystem debugging playbook, but “one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.” The removal took out servers supporting two key S3 subsystems — the index subsystem (“manages the metadata and location information of all S3 objects”) and the placement subsystem (“manages allocation of new storage”) — both of which required a full restart. S3 in us-east-1 was disrupted from roughly 9:37 AM to 1:54 PM PST (~4 hours 17 minutes). Cascading: a huge fraction of the internet that depended on S3 went down (Slack, Quora, Trello, Imgur, Dropbox’s then-S3-backed Magic Pocket — see Dropbox File Sync System Design — and many more). The remediation: AWS modified the tool to “remove capacity more slowly” and to refuse removals that would take any subsystem below its minimum required capacity, audited other operational tools for similar safeguards, and re-partitioned the index subsystem into smaller “cells” to shrink restart time and blast radius.

10. Tradeoffs

Decision	Choice	Alternative	Why
Storage redundancy	Erasure coding (1.4× overhead)	3-replica (3× overhead)	Massive cost saving at exabyte scale
Consistency (pre-2020)	Eventual for overwrites and LIST	Strong	Performance + simplicity, given the indexing tech of the era
Consistency (2020+)	Strong for all ops	Eventual	Customer pain with eventual; possible because of new witness architecture
API	Flat key-blob (object)	Hierarchical filesystem (GFS)	Simpler at internet scale; HTTP-native
Bucket namespace	Globally unique	Per-account	URL routing simpler; cost is global serialization on bucket create
Scaling unit	Per-prefix	Per-bucket	Hot prefixes auto-shard without operator action
Cold tier	Glacier (separate class)	Same hardware, different SLA	Different physical medium for radically lower cost
Multipart upload	Required for objects > 5 GB	Single PUT	Resume on failure; parallel upload
Versioning	Opt-in per bucket	Always on	Backwards-compat with original API
Cross-region replication	Async	Synchronous	Sync would tie GET latency to slowest region; async meets most use cases

11. Pitfalls

1. Pre-2020 eventual consistency on overwrites. The single most-cited gotcha. Code that PUT-then-GET on the same key would intermittently see old bytes for seconds. Many production bugs traced to this. Fixed by the December 2020 strong-consistency rollout.

2. LIST-after-write inconsistency (also fixed 2020). A PUT followed by LIST might not include the new key for some seconds; downstream pipelines that LIST’d to find new files would silently skip them.

3. Hot prefix throttling. Pre-2018, all keys with the same prefix went to one shard; sustained writes to 2023/01/01/... keys would 503 (“SlowDown”) under load. Mitigation was random-prefix workaround; auto-resharding since 2018 mostly removes this.

4. The 5 GB single-PUT limit. Larger objects require multipart upload. A naive client that tries to PUT a 10 GB file gets a 400 error.

5. Multipart upload abandonment leaks storage. Started multiparts that never call CompleteMultipartUpload consume storage indefinitely. Lifecycle rule AbortIncompleteMultipartUpload (set to 7 days) is recommended on every bucket.

6. Pre-signed URL leakage. A pre-signed URL is a bearer credential — anyone with the URL can act on it. Logging URLs to access logs, embedding in emails, posting in chat, and so on can leak access. Set short expiry times (minutes, not days) and avoid placing them in logs.

7. Cross-Region Replication is async. A PUT to bucket-A in us-east-1 takes potentially seconds-to-minutes to appear in bucket-B in eu-west-1. Disaster-recovery designs that assume synchronous replication will silently lose data on regional failover.

8. Bucket name squatting and the global namespace. Because bucket names are globally unique, “your-company.s3.amazonaws.com” can be claimed by anyone first. Subdomain-takeover attacks exploit dangling DNS records pointing at deleted S3 buckets that an attacker re-creates.

9. Public bucket misconfiguration. Bucket policies that grant s3:GetObject to Principal: "*" make every object publicly readable. The 2017 wave of breaches (Verizon, Accenture, RNC) was almost entirely this mistake. AWS now enables “Block Public Access” by default at the account level.

10. Listing a bucket with millions of keys is slow. LIST returns 1000 keys per page; iterating through 100 million keys takes 100,000 round trips. Inventory reports (a daily flat-file dump of the bucket’s keys, generated asynchronously) is the right answer for batch enumeration at scale.

11. Eventual consistency of access-control changes. Bucket policies and IAM changes propagate eventually. A test that fails right after granting permission might succeed seconds later. Some replication of permissions is faster than others.

12. Common Interview Variants

• “Design Amazon S3” — the canonical full-system question. Hit: HTTP API, bucket+key, erasure coding for durability, metadata index sharded by key, multi-AZ replication, eventual-vs-strong consistency story, Glacier tier.
• “Design Dropbox” — see Dropbox File Sync System Design — overlaps in deduplication and cold storage but has the sync protocol layer on top.
• “How does S3 achieve 11 nines of durability?” — erasure coding + multi-AZ + fast repair. Walk through the math.
• “Why was S3 eventually consistent for so long, and how did they convert it to strong?” — explain the witness-service architectural addition; emphasize that consistency upgrades on a deployed exabyte-scale system are nontrivial.
• “What happens if you PUT then GET on the same key?” — strongly consistent since 12/2020; before that, eventual for overwrites.
• “Design an object store on commodity hardware” — Ceph or MinIO style. Erasure coding, gossip-based membership, CRUSH or consistent hashing for placement.
• “How does S3 handle a hot key?” — the underlying index auto-reshards hot prefixes; for a single hot object, range-GET fanout from clients distributes load across shards.
• “What are pre-signed URLs and when would you use them?” — token-based time-bounded access; offload upload/download from your backend.
• “Why is multipart upload required for large objects?” — to enable parallel upload, partial retry, and to bound per-request memory on the server.

13. Open Questions / Uncertain

• Exact erasure-coding parameters used by S3 (k, m), and whether intra-AZ redundancy is EC or replication. Industry rumor: variants of (10, 4) and (6, 3); not officially confirmed.
• The witness mechanism introduced in 2020 is now known to be a cache-coherence read barrier (per Vogels 2021), but the wire protocol, quorum, and failure semantics remain undocumented.
• Whether Glacier physically uses tape or simply spun-down disk; AWS has been deliberately ambiguous.
• How much of the metadata layer is built on DynamoDB vs an internal predecessor.

14. See Also

• Google File System Design — file-system cousin; different API and consistency model
• Distributed Key Value Store System Design — what the metadata index is, internally
• Distributed Log System Design — common downstream for S3 event notifications
• Dropbox File Sync System Design — built atop an S3-compatible store originally; migrated to Magic Pocket
• Bloom Filter — plausible negative-cache for prefix listing (not officially documented)
• Consistent Hashing — placement strategy in MinIO and Ceph (not the same as S3’s auto-resharding)
• Content Delivery Network System Design — CloudFront sits in front of S3 for caching
• Major System Designs MOC
• SWE Interview Preparation MOC