vault

Amazon E-Commerce Order Pipeline System Design

21 min read

Amazon E-Commerce Order Pipeline System Design

An e-commerce order pipeline is the canonical case study for microservice orchestration over heterogeneous storage at planetary scale. From “user clicks Add to Cart” to “package arrives at the door,” a single order traverses dozens of services — catalog, search, cart, pricing, inventory, payment, fraud, fulfillment, shipping, returns — each owning its own data store, deployed independently, scaled independently. The order is not a single database transaction; it’s a long-running saga coordinated across services that may fail, retry, and compensate. The lineage of this architecture traces directly to two foundational artifacts: the Bezos API mandate of 2002 (every team must expose its functionality only via service interfaces, no shared databases, all interfaces designed as if external) and the Dynamo paper (DeCandia et al. SOSP 2007) which formalized eventual consistency, Consistent Hashing for sharding, and quorum-based replication for the underlying KV store. This note develops the order pipeline from cart through fulfillment, citing the original sources, and treating Amazon as the case study with eBay’s Buy-It-Now and large e-commerce comparators (Shopify, Walmart) as variants.

1. Requirements

Functional Requirements

  • Catalog browsing. Hundreds of millions of products across thousands of categories. Users browse by category, search by keyword, refine by filters (brand, price, rating, Prime-eligible).
  • Search. Full-text search across product titles and descriptions, plus structured filters (price range, rating, brand). Query understanding (synonyms, typos), ranking by relevance + popularity + personalization.
  • Cart. A user’s shopping cart persists across sessions and devices. Items can be added, modified, removed.
  • Pricing. Per-product price varies by region, by promotion, by customer (Prime discount), and dynamically by demand. Tax calculation per jurisdiction.
  • Checkout. User selects shipping address, shipping method, payment method; system computes total (subtotal + tax + shipping − promotions) and creates the order.
  • Inventory reservation. Across multiple fulfillment centers (FCs), the system reserves units of each product to fulfill the order.
  • Payment. Charges the user’s card or account credit; integrates with multiple payment processors.
  • Order fulfillment. A multi-stage workflow: order accepted → assigned to FC → picked → packed → shipped → in transit → delivered.
  • Tracking. User sees real-time order status; receives notifications at major state transitions.
  • Returns. User initiates return; system generates label, tracks return shipment, processes refund on receipt.
  • Recommendation. Personalized product recommendations on home page, product detail page, post-purchase emails.
  • Fraud detection. Real-time scoring of every order; high-risk orders held for manual review.

Non-Functional Requirements

  • Catalog scale. ~600M products listed worldwide (Amazon’s catalog is estimated to exceed this; exact number not public).
  • Search scale. Tens of billions of search queries per day across all Amazon properties. Search QPS dominates everything else.
  • Peak load. Prime Day and Black Friday/Cyber Monday produce 5–10× normal load. Capacity planning is the annual engineering exercise.
  • Order rate. Hundreds of orders per second average; thousands per second peak. Cumulative billions of orders historically.
  • Multi-region. Operates in dozens of countries with regulatory, currency, and tax variations. Latency budget: <200 ms for any user-facing read in their region.
  • Availability. Catalog/search ≥ 99.99%; checkout ≥ 99.99% (a five-minute checkout outage on Prime Day is a catastrophic revenue event); fulfillment systems ≥ 99.95%.
  • Consistency. Cart can be eventually consistent — losing a cart item occasionally is acceptable. Inventory must converge to correct under reconciliation. Payment + order placement must be strongly consistent (no double-charge, no double-fulfill).
  • Durability. Order records: 11 nines (effectively forever). Catalog data: high durability. Cart: lower (acceptable to lose, since users will re-add).

2. Capacity Estimation

  • Catalog: ~600M products. Each product document: ~5 KB (title, description, category, attributes, image refs, ratings, etc.). Total catalog text: ~3 TB. Photos and videos are an order of magnitude larger and live in object storage.
  • Search index: Inverted index over product text + structured fields ≈ ~5–10 TB sharded across an Elasticsearch/Lucene-like cluster of hundreds of nodes.
  • Daily search QPS: Conservatively 1B searches/day → 12K QPS average, peak 100K QPS+.
  • Daily orders: Tens of millions globally. Peak hour during Prime Day: hundreds of thousands of orders/hour ≈ ~100/s sustained, with bursts to 1000+/s.
  • Order record size: ~10 KB (line items, addresses, payment refs, fulfillment refs, status log, history).
  • Cart record size: ~2 KB per active cart; ~50M active carts at any moment ≈ 100 GB.
  • Inventory state: ~600M products × ~100 fulfillment centers worldwide × ~50 B per (product, FC) row = 3 TB. Sharded by product_id or (product_id, fc_id).
  • Bandwidth: Image and video CDN egress dominates by far — many tens of petabytes per month at this scale.

The big insight: search and catalog reads massively dominate write QPS. Architecture biases reads to caches and inverted indexes; writes go to a small set of authoritative services with strong consistency.

3. API

GET    /v1/products
  Query: ?q=&category=&filters=&page=
  → 200 { products: [{id, title, price, image, rating, prime}], facets, next }

GET    /v1/products/{product_id}
  → 200 { id, title, description, images, attributes, price, stock_status,
          recommendations, reviews }

POST   /v1/cart/items
  Body: { product_id, qty }
  → 200 { cart: {...} }

GET    /v1/cart
  → 200 { items: [{product_id, qty, price}], subtotal }

POST   /v1/checkout
  Body: { cart_token, shipping_address_id, shipping_method, payment_method_id,
          promo_code?, idempotency_key }
  → 201 { order_id, total, eta }

GET    /v1/orders/{order_id}
  → 200 { id, items, status, eta, tracking_url, fulfillment_state }

POST   /v1/orders/{order_id}/cancel
  → 200 { refund: {amount, status} }

POST   /v1/returns
  Body: { order_id, items: [{order_item_id, qty, reason}] }
  → 201 { return_id, label_url }

GET    /v1/recommendations
  Query: ?context=home|pdp|cart&product_id?
  → 200 { products: [...] }

4. Data Model

4.1 Catalog

products:                         # eventually consistent KV (Dynamo-style)
  product_id  PRIMARY KEY (string)
  title, description, brand, category_path
  attributes JSON  (variant axes — size, color, ...)
  images []
  base_price, currency
  active BOOL
  last_updated_at

product_variants:                 # specific SKU per (product, attribute combo)
  sku, product_id, attributes, price, weight, dimensions

categories:
  category_id, parent_id, name, attribute_schema

The catalog is read-heavy and mostly eventually consistent — a price update propagating in a few seconds is fine. Stored in a Dynamo-derived KV store; Werner Vogels’s Eventually Consistent (2008) explicitly cites the catalog as the canonical eventual-consistency case at Amazon.

4.2 Cart

carts:                            # session-affinity KV
  cart_id          PRIMARY KEY
  user_id          (nullable for guests)
  items JSON        ([{product_id, qty, added_at}, ...])
  last_updated_at

The Dynamo paper explicitly used the shopping cart as its prime example of why eventual consistency is acceptable. Cart updates merge via vector clocks: when two replicas of a cart diverge (e.g., the user adds item A on phone and item B on web simultaneously), the merge is “union of items” — the customer might see both, which is mildly annoying but never destructive (worst case: customer removes the unwanted item at checkout). The alternative — strong consistency — would require coordination on every cart write, defeating the latency budget.

4.3 Inventory

inventory:
  product_id, fc_id (fulfillment center), available_qty, reserved_qty,
  inbound_qty, last_updated_at
  PRIMARY KEY (product_id, fc_id)

Sharded by product_id. Updates happen on order placement (decrement available, increment reserved), shipment (decrement reserved), and inbound restock (increment available, decrement inbound). Reconciliation runs nightly against actual physical counts at each FC.

Inventory across FCs is logically eventually consistent — a single global product available_qty is the sum across FCs, computed asynchronously. The consistency boundary is the per-FC row.

4.4 Orders

orders:                           # strongly consistent
  order_id          BIGINT PK
  user_id           BIGINT INDEX
  state             ENUM (pending, paid, allocated, picking, packed, shipped, delivered, canceled, refunded)
  items JSON        ([{product_id, qty, price, fc_id, status}, ...])
  shipping_address  JSON
  payment_ref       TEXT
  total             DECIMAL
  fraud_score       DECIMAL
  idempotency_key   UUID UNIQUE
  state_log JSON    (saga step audit)
  created_at, updated_at

order_state_history:              # immutable append-only event log
  event_id, order_id, event_type, payload JSON, occurred_at

Sharded by user_id for the “my orders” view; orders also indexed by FC for fulfillment querying.

4.5 Search Index

Inverted index over product titles, descriptions, attributes; built on Lucene-style segment files using LSM Tree storage internally. Updated via Distributed Log System Design CDC from the catalog KV.

5. High-Level Architecture

flowchart TB
    subgraph Edge
        APIGW[API Gateway]
        CDN[CDN<br/>(images + static)]
    end
    subgraph Discovery
        Search[Search Service]
        ES[(Inverted Index<br/>Lucene-based)]
        Rec[Recommendation Service]
        Catalog[Catalog Service]
        CatStore[(Dynamo KV)]
    end
    subgraph CartTier["Cart"]
        CartSvc[Cart Service]
        CartStore[(Dynamo KV)]
    end
    subgraph CheckoutSaga["Checkout (Saga)"]
        OrderSvc[Order Service]
        OrderDB[(Sharded order DB)]
        SagaOrch[Saga Orchestrator]
        Pricing[Pricing + Tax]
        Fraud[Fraud Service]
        Inv[Inventory Service]
        InvDB[(Sharded inventory DB)]
        PaySvc[Payment Service]
        PSP[(Card networks / Amazon Pay)]
    end
    subgraph Fulfillment
        FulfilOrch[Fulfillment Orchestrator]
        WMS[Warehouse Mgmt]
        Shipping[Shipping Carrier Integration]
    end
    subgraph Comms
        Notif[Notification Service]
    end
    subgraph Async
        Kafka[Kafka / Kinesis]
        DW[(Data Warehouse / S3 + Glue)]
    end

    APIGW --> Search
    Search --> ES
    Search --> Rec
    APIGW --> Catalog
    Catalog --> CatStore
    APIGW --> CartSvc
    CartSvc --> CartStore
    APIGW --> OrderSvc
    OrderSvc --> SagaOrch
    SagaOrch --> Pricing
    SagaOrch --> Fraud
    SagaOrch --> Inv
    SagaOrch --> PaySvc
    Inv --> InvDB
    PaySvc --> PSP
    OrderSvc --> OrderDB
    OrderSvc --> FulfilOrch
    FulfilOrch --> WMS
    FulfilOrch --> Shipping
    OrderSvc --> Notif
    OrderSvc --> Kafka
    Inv --> Kafka
    Catalog --> Kafka
    Kafka --> ES
    Kafka --> Rec
    Kafka --> DW

What this diagram shows. The architecture is canonically microservice-shaped, descended from the Bezos API mandate. Discovery (Search, Catalog, Recommendations) is read-heavy; the Catalog Service owns its KV store, Search owns its inverted index, and they communicate via service APIs (no shared DB). The Cart Service is its own bounded context with its own Dynamo-style store. Checkout invokes a Saga Orchestrator that drives a multi-step distributed transaction across Pricing, Fraud, Inventory, and Payment — each step’s failure triggers compensating actions. The Fulfillment subsystem is a separate workflow that takes over after order placement. Async events flow into Kafka for downstream consumers (search index sync, recommendations training, analytics). Crucially, no service owns another service’s data — even the Order DB stores only references to inventory, payments, and fulfillment, never copies of their data.

6. Request Flow — Place Order

sequenceDiagram
    participant U as User
    participant GW as API Gateway
    participant Cart as Cart Svc
    participant Order as Order Svc
    participant Saga as Saga Orchestrator
    participant Price as Pricing+Tax
    participant Fraud as Fraud
    participant Inv as Inventory
    participant Pay as Payment
    participant Fulfil as Fulfillment
    participant Notif as Notification

    U->>GW: POST /v1/checkout {cart_token, addr, payment, idempotency_key}
    GW->>Order: create_order
    Order->>Order: dedupe by idempotency_key
    Order->>Saga: start

    Saga->>Cart: read cart items
    Cart-->>Saga: items
    Saga->>Price: compute totals(items, addr, promo)
    Price-->>Saga: subtotal, tax, shipping, total
    Saga->>Fraud: score(user, payment, items, ip, address)
    Fraud-->>Saga: ok / hold-for-review
    alt fraud_hold
        Saga->>Order: state=pending_review
        Saga-->>U: 201 (will email when reviewed)
    else ok
        Saga->>Inv: reserve(items, region) → choose FCs
        Inv->>Inv: per-(product, fc) CAS:<br/>UPDATE available -=q, reserved += q WHERE available >= q
        Inv-->>Saga: reservations[]
        Saga->>Pay: authorize(amount)
        Pay-->>Saga: hold_id
        Saga->>Order: state=paid, persist allocations
        Saga->>Fulfil: create fulfillment work
        Fulfil-->>Saga: work_id
        Saga->>Notif: send confirmation email
        Saga-->>U: 201 {order_id, eta}
    end

Reading the sequence. The saga sequences seven distinct sub-operations across services. Each step has explicit failure handling: if Inventory cannot reserve any item, compensate by canceling already-reserved items and aborting; if Payment fails, compensate by un-reserving inventory; if any step throws, the saga state is persisted (in the saga orchestrator’s own DB, often using Cadence/Temporal or an Amazon-internal equivalent) so a coordinator restart resumes mid-saga. The Inventory reserve step is itself a distributed CAS: the system selects target FCs for each item (close to user, with stock), then atomically updates each (product, fc) row to convert available → reserved. If a row’s CAS fails (someone else got the last unit), the saga retries with a different FC or fails the line item.

7. Deep Dive 1 — Dynamo and Eventual Consistency

The Dynamo paper (DeCandia et al., SOSP 2007) is the foundational document. It introduced (or popularized) the cluster of techniques used by every “big-table-style” KV store since:

  • Consistent Hashing with virtual nodes for partitioning. Each key’s preference list is the next N successor nodes on the ring.
  • Quorum reads/writes: for an N-replica setup, R + W > N guarantees overlap; tunable consistency (e.g., N=3, W=2, R=2 gives strong-ish consistency; N=3, W=1, R=1 gives high availability with eventual consistency).
  • Vector clocks for reconciling concurrent updates: when two clients write to the same key at different replicas, the system can detect the conflict and either auto-merge (e.g., set union for shopping carts) or surface both versions to the application.
  • Hinted handoff for handling temporary node failures: if a target replica is down, write goes to a substitute node with a “hint” to forward when the original recovers.
  • Merkle-tree anti-entropy for detecting and repairing diverged replicas.
  • Gossip-based membership.

The Dynamo paper’s running example is the shopping cart. Vogels’s Eventually Consistent (ACM Queue 2008) elaborates: a cart is a CRDT-like merge (“set of items added”), and even if a customer briefly sees an old cart they recover trivially.

Modern AWS DynamoDB is the commercial successor — different implementation but same principles. Amazon’s internal services use a mix of DynamoDB, custom Dynamo-derivatives, and S3 (which itself moved to strong read-after-write consistency in 2020 as a notable architectural shift).

8. Deep Dive 2 — Inventory and the Saga Pattern

The order checkout illustrates why Two-Phase Commit is unsuitable at this scale and shape. 2PC requires:

  • All participants to support the prepare/commit protocol.
  • A single coordinator with knowledge of all participants.
  • Locks held across the prepare-to-commit window.
  • All participants live throughout the commit.

In a microservice world where Payment is an external SaaS (sometimes Stripe), Inventory is sharded across hundreds of databases, Fraud is a stateless service, Fulfillment is a long-running workflow, none of this fits. The saga (Garcia-Molina & Salem 1987) is the alternative: a sequence of local transactions, each with a defined compensating transaction, executed in order. Failures trigger backward compensation.

Pat Helland’s Life Beyond Distributed Transactions (CIDR 2007) — written while at Amazon — is the foundational engineering essay arguing that distributed transactions don’t scale and that idempotent + retryable + sagas are the path forward. It directly informs Amazon’s design.

AWS now publishes this pattern verbatim as canonical guidance: the AWS Prescriptive Guidance — Saga orchestration pattern names exactly three example participants — order service, inventory service, payment service — with compensating transactions Revert Payment, Revert Inventory, Remove Order. AWS recommends AWS Step Functions (standard workflow) as the orchestrator specifically because “Step Functions mitigates the single point of failure issue, which is inherent in the implementation of the saga orchestration pattern. Step Functions has built-in fault tolerance and maintains service capacity across multiple Availability Zones in each AWS Region to protect applications against individual machine or data center failures.” (per AWS Prescriptive Guidance). The same document enumerates the saga’s enduring concerns — idempotency, semantic locking for isolation, compensating-transaction latency, observability, eventual consistency — that any production order pipeline must engineer around.

For the order pipeline, the saga is roughly:

StepForward actionCompensation
1Read cart(none)
2Compute pricing(none)
3Score fraud(none)
4Reserve inventory across FCsUn-reserve inventory
5Authorize paymentVoid hold
6Persist order, state=paidMark canceled
7Trigger fulfillmentCancel fulfillment work

Idempotency is built in at every step via the saga’s idempotency_key. The state of the saga itself is persisted (e.g., in Amazon SWF, AWS Step Functions, or a Cadence/Temporal instance) so a coordinator crash recovers mid-saga. The AWS Prescriptive Guidance specifically calls out that “saga participants need to be idempotent to allow repeated execution in case of transient failures caused by unexpected crashes and orchestrator failures.”

9. Deep Dive 3 — Fulfillment Workflow

Fulfillment is its own multi-stage workflow, much longer-lived than the checkout saga (hours to days):

order accepted
   → assigned to FC (based on stock + shipping cost + carbon)
   → picked (warehouse worker pulls items from shelves)
   → packed (boxed, weighed, labeled)
   → shipped (handed off to carrier)
   → in transit (carrier API events)
   → out for delivery
   → delivered (or returned if undeliverable)

Each transition is an event in Kafka; each is a workflow step with timeouts and retries. The Warehouse Management System (WMS) is itself a complex application coordinating physical robotics (Amazon’s Kiva), barcode scanners, and human workers. From the order pipeline’s perspective, WMS is a black box that emits “picked” / “packed” events.

The interaction with carrier APIs (UPS, FedEx, USPS, Amazon Logistics) is webhook-based: each provides shipment status updates that the order pipeline ingests, normalizes, and surfaces to the user as tracking. See Webhook Delivery System Design for the at-least-once delivery semantics.

The fulfillment workflow runs on something like AWS Step Functions or a Distributed Task Scheduler System Design platform (Cadence, Airflow, Temporal). Failure handling is intricate: a damaged item discovered at picking triggers a different sub-workflow (substitute or refund). Backorder handling, partial shipments, and split orders (different items shipped from different FCs) all branch the state machine.

10. Scaling

10.1 Sharding

  • Catalog: sharded by product_id hash via Consistent Hashing. Reads are dominated by the search index (which is itself sharded); direct catalog reads are mostly batched fetches per product page.
  • Inventory: sharded by (product_id, fc_id). A booking that touches multiple products and multiple FCs hits multiple shards — but each shard transaction is independent (no cross-shard transactional commit; the saga handles the global atomicity).
  • Orders: sharded by user_id for the “my orders” view; secondary indexed by FC for fulfillment.
  • Cart: sharded by user_id (or cart_id).
  • Search: sharded by random hash of product_id; replicated for read parallelism and tail-latency hedging.

10.2 Caching

  • Product detail page: heavily cached (CDN + per-region service cache). TTL ~minutes; invalidate on price change or stock-out via Kafka events.
  • Search results: cacheable for popular queries (e.g., “iphone case”) for short TTLs.
  • Cart: read-through cache in the cart service; the cache is the working state, persisted to DynamoDB as the source of truth.

10.3 Pre-Scaling for Peak Events

Prime Day and Black Friday/Cyber Monday are the annual capacity event. Pre-scaling is months in advance: capacity reviews, load testing at projected peak, soak testing, incident response rehearsals. Amazon famously uses Prime Day to stress-test infrastructure that AWS later sells to customers.

10.4 Multi-Region

Each major geography has its own catalog, search, cart, order, and fulfillment stack. User profile data has global replication for cross-region login. Fulfillment is regional by physics — packages ship from FCs in the same country. Cross-region inventory reservation (e.g., shipping from another country) is rare and special-cased.

10.5 The Event Bus

Kafka (or Amazon-internal Kinesis equivalents) is the asynchronous backbone. Events: OrderPlaced, InventoryDecremented, OrderShipped, OrderDelivered, OrderRefunded, ProductUpdated. Consumers: search index sync, recommendation training, fraud retraining, analytics, accounting, customer email triggers. Kafka decouples producers from the (constantly growing) set of downstream consumers — this is precisely the architectural lever the Bezos mandate enabled.

10.6 Storage Foundations: LSM Trees

Both DynamoDB internally and many of the supporting stores (catalog, KV) use LSM Tree (Log-Structured Merge-Tree) storage engines, well-suited to high write throughput with reasonable read amplification. The original LSM paper (O’Neil et al. 1996) predates Dynamo but underpins much of the modern ecosystem.

11. Real-World Example

Amazon’s Documented Architecture

  • The Bezos API Mandate (2002). Per Steve Yegge’s internally-leaked-then-published rant, Bezos sent an internal memo mandating: every team must expose its functionality via service interfaces only, no exceptions, no shared databases or back doors, all interfaces designed to be externalizable. This single decision created the conditions for both the modern microservice paradigm and AWS itself (since the service interfaces could be opened to external customers).
  • Dynamo (SOSP 2007). The KV store underlying many Amazon services. The 2007 paper documented the shopping cart, session store, and best-seller list among its production users.
  • AWS DynamoDB (launched 2012). Commercial follow-on to Dynamo, with many architectural differences (single-region strong consistency by default; later cross-region replication via Global Tables).
  • Amazon S3 strong consistency (2020). A notable architectural inflection — for many years S3 was eventually consistent for overwrite-PUT and DELETE; the 2020 announcement introduced strong read-after-write consistency without API changes, demonstrating the practical importance of strong consistency for object stores.
  • AWS Step Functions, SWF. Workflow orchestration services that look very much like internal saga orchestrators productized for external customers.
  • Amazon Personalize, Amazon Forecast, etc. ML services that started as internal recommendation/forecasting systems.

Comparable Platforms

  • eBay. Auctions add dynamic-bid logic on top of the e-commerce skeleton; Buy-It-Now is structurally similar.
  • Shopify. Multi-tenant e-commerce platform; each merchant is a tenant with their own catalog/orders/fulfillment, sharing infrastructure. Dispatch through Kafka, Rails-based monolith decomposing toward microservices.
  • Walmart.com. Direct competitor; significant engineering investment in the past decade. Hybrid of legacy systems and modern microservices.
  • Alibaba. Singles’ Day generates the world’s largest e-commerce peak (multiple times Prime Day’s peak); their architecture leans heavily on flash-sale patterns and very deep Kafka usage.

Uncertain

Verify: Amazon’s current (post-2015) internal architectural specifics — the actual choreography of checkout, the orchestrator engine used internally, the per-FC inventory CAS protocol, fraud-model architecture. Reason: Amazon does not publish post-Dynamo internals; the descriptions above synthesize the well-attested primary sources (SOSP 2007 Dynamo paper for the KV substrate, Vogels’s 2008 Eventually Consistent for the cart-merge example, the Bezos 2002 mandate via Yegge’s leak, the 2020 S3 strong-consistency announcement, Helland CIDR 2007 for the distributed-transaction philosophy, and AWS Prescriptive Guidance for the externalized saga pattern AWS sells to customers) with inference from analogous AWS-productized services (Step Functions, DynamoDB, OpenSearch, EventBridge). The shape of the architecture (microservice mandate + saga + eventual consistency + per-FC sharding) is high-confidence; specific module boundaries and orchestrator choices are low-confidence. To resolve: any future re:Invent talk or Amazon engineering blog disclosing a specific internal pipeline; the most useful would be a post-2020 follow-up to the original Dynamo paper.

12. Tradeoffs

DecisionOption AOption BWhen to pick AWhen to pick B
Distributed transactionTwo-Phase CommitSaga + idempotencySingle tightly-coupled DBMicroservices over heterogeneous storage
Cart consistencyStrongEventual + vector clock mergeSmall scaleMassive scale (B is the Dynamo lesson)
Catalog consistencyStrongEventualReal-time pricingMost catalog reads (B)
SearchPostgres full-textLucene / inverted indexTiny catalogHundreds of millions of products
Inventory modelSingle globalPer-FC shardedSingle warehouseMulti-FC (B is the standard)
Workflow engineCron + scriptsStep Functions / CadenceFew simple workflowsThousands of multi-day workflows
RecommendationRule-basedML candidate gen + rankingFew productsHundreds of millions
CommunicationSynchronous RPCAsync events (Kafka)Few servicesMany independent teams (B; Bezos mandate logic)

13. Pitfalls

  1. Trying to build a monolithic order DB at scale. The Bezos lesson: shared DBs don’t scale across teams. Each service owns its own data; communication is via APIs and events.
  2. No idempotency on POST /v1/checkout. A retry duplicates the order. Always require idempotency_key; dedupe at the API edge.
  3. Cart with strong consistency. Strong consistency on cart writes serializes shopping across user devices; eventual + merge is correct for the use case.
  4. Inventory reserved without TTL. Abandoned reservations freeze stock. Use a saga timeout — if checkout doesn’t complete in N minutes, un-reserve.
  5. Search not driven by CDC. A direct write to search on every catalog change couples services. CDC via Kafka is the loosely-coupled path.
  6. Single global inventory counter. Cross-FC contention destroys throughput. Per-FC sharding is mandatory.
  7. Synchronous fulfillment. Don’t make checkout block on fulfillment readiness — it’s a long-running workflow.
  8. Ignoring split orders. A single order may ship from multiple FCs; design state model to support per-line-item state.
  9. Treating refunds as undoing the order. A refund is its own first-class workflow with its own state machine; an “order” once placed is immutable, and refunds/returns are events on top.
  10. No fraud holding. Without a hold-for-review path, every fraudulent order ships before review completes.
  11. Underestimating tail latency on payment processors. PSPs occasionally take seconds; the saga must tolerate this with retry + circuit breaker, not block the user UI.
  12. Forgetting to compensate at every step. A half-rolled-back saga leaves inventory reserved with no order — silent data corruption.

14. Common Interview Variants

  • “Design Amazon.” Canonical prompt. Walk through search, cart, checkout saga, inventory, fulfillment.
  • “Design eBay.” Adds auctions — a bidding system with sniping, anti-shill, automatic bids.
  • “Design Shopify.” Multi-tenancy emphasis — many merchants on one platform, isolation, customizable storefronts.
  • “Design a flash-sale system.” Add a virtual waiting room (see Ticketmaster Booking System Design) on top.
  • “Design a recommendation system.” Subset; see Recommendation Engine System Design.
  • “Design a search engine for products.” Subset; see Distributed Search System Design.
  • “Design a payment system.” Subset; payment processing, reconciliation, double-charge prevention.
  • “Design Uber’s fulfillment network for groceries.” Variant — same shape as Amazon’s last-mile but with hyperlocal sourcing.

15. Open Questions

  • How does Amazon’s modern internal stack compare to the 2007 Dynamo paper? Public information is extremely sparse post-2015.
  • What is the actual fraud-hold rate in production, and how is the model retrained against shifting fraud patterns?
  • How does Amazon handle inventory truth vs catalog truth at the FC physical level — given inventory drift between recorded and actual, how often is the gap reconciled?
  • What was the migration path for S3 to strong consistency in 2020 without API changes? The post claims no perf impact; the engineering achievement is significant but under-documented.
  • How do split shipments and partial fulfillments interact with payment timing — is the customer charged on the first shipment, the last, or per-shipment?

16. See Also

Graph View