API Gateway System Design

An Application Programming Interface (API) Gateway is a single-entry-point reverse proxy that sits between external clients (mobile apps, browsers, third-party integrators) and an internal mesh of backend microservices. It terminates Transport Layer Security (TLS), authenticates the caller, enforces per-tenant quotas, routes the request to the correct upstream, transforms request and response payloads, and emits observability data — all before the application logic runs. The pattern emerged from the practical reality that a microservice fleet of fifty or five hundred services cannot each independently re-implement the cross-cutting concerns of authentication, rate limiting, retries, and metrics without diverging into chaos. The canonical industrial deployments are Netflix Zuul (Java; gateway for the Netflix streaming product), Kong (Lua plugin engine on top of nginx; the open-source de facto standard), AWS API Gateway (managed service), Apigee (Google Cloud’s enterprise offering, originally founded 2004), Tyk, Spring Cloud Gateway, and increasingly Envoy used as a gateway (the same data plane that powers Istio’s service mesh). This note covers the design choices an interviewer expects: request lifecycle, plugin architecture, rate limiting, hot reloading of configuration, and the BFF (Backend For Frontend) variant that arises when one gateway is too monolithic to serve every client class.

1. Why This System Appears in Interviews

Three reasons make API Gateway a fixture of senior-level system design rounds. First, it is the integration seam of every microservice architecture; understanding it requires fluency in TLS, OAuth/JWT, rate-limiting algorithms, circuit breakers, retries, and observability all at once — a small set of topics each on their own, but the gateway forces you to compose them. Second, the latency budget is brutal — an interviewer can probe how you keep gateway overhead under five milliseconds while still doing five plugin steps per request, and that probe spans data-structure choice (in-memory hash maps vs Redis), caching design (token introspection caching), and concurrency model (event loop vs thread pool). Third, the gateway is a load-bearing single point of failure in production; the interviewer can pivot from “design the happy path” to “the gateway runs out of file descriptors at peak — walk me through what you do,” which is where you demonstrate operational maturity around connection pooling, graceful drain, and blue/green deployment of gateway instances themselves.

The interview pattern is less “build a gateway from scratch” (which would take weeks) and more “design the request lifecycle, the plugin chain, the configuration plane, and the failure modes.” This note is structured accordingly.

2. Requirements

2.1 Functional Requirements

1. Request routing. Match an incoming HyperText Transfer Protocol (HTTP) or gRPC (Google Remote Procedure Call) request to one of N backend services based on path (/users/* → user-service), host header (api.example.com vs internal.example.com), method, headers, or arbitrary expressions. Apply path rewrites (/api/v1/users/123 → /users/123 upstream).
2. Authentication. Verify caller identity via API keys (a static token in a header), JSON Web Tokens (JWT, RFC 7519 — signed claims), OAuth 2.0 access tokens (RFC 6749 — introspection or local validation), or mutual TLS (mTLS — client certificate validation). Reject unauthenticated requests early.
3. Authorization. Enforce that the authenticated caller is permitted to invoke the matched route — usually role-based or scope-based, sometimes attribute-based.
4. Rate limiting and quota. Cap requests per consumer per time window (e.g., 1000 requests/minute per API key). Return 429 Too Many Requests with a Retry-After header when exceeded. Support multiple dimensions: per-key, per-IP, per-route, per-organization.
5. Request and response transformation. Inject or strip headers, rewrite paths, transform payloads (XML ↔ JSON), aggregate multiple upstream responses into one client response (the request-aggregation pattern, useful for mobile clients on slow networks).
6. Caching. Cache idempotent (typically GET) responses keyed by URL plus a configurable subset of headers, with TTL.
7. Circuit breaking and retries. Protect upstreams: trip a circuit breaker after a threshold of failures and short-circuit subsequent requests to the same upstream until a half-open probe succeeds. Retry transient failures with backoff (capped to avoid retry storm).
8. Observability. Emit structured logs, metrics (request rate, error rate, latency histograms — the RED method: Rate, Errors, Duration), and distributed-tracing spans for every request.
9. Multi-tenancy. Isolate consumers from each other; one tenant’s traffic spike must not starve another’s quota.
10. Hot configuration reload. Push new routes, plugins, or rate-limit rules without dropping in-flight connections or restarting the process.

2.2 Non-Functional Requirements

1. Latency overhead. p99 added latency from the gateway < 5 ms above the upstream’s own latency, in the steady state. p50 ideally < 1 ms.
2. Throughput. Each gateway instance sustains at least 100,000 queries per second (QPS) at small request size on commodity hardware. Real-world deployments scale by adding instances behind a load balancer (see Load Balancer System Design).
3. Availability. 99.99% on the gateway tier. The gateway is on the critical path of every external request; it must be more available than the services behind it.
4. Connection scalability. Tens of thousands of concurrent persistent connections per instance (long-lived mobile clients, server-sent events, WebSocket upgrades).
5. Configuration scale. Tens of thousands of routes and hundreds of plugins per instance, with sub-second propagation of config changes.

Design budget, not a benchmark

The 100K QPS per instance figure is an order-of-magnitude design budget, not a benchmark of any specific product. Envoy’s own performance FAQ states plainly that “there is no single QPS, latency or throughput overhead that can characterize a network proxy such as Envoy” and that any number must be measured contextually (Envoy benchmarking FAQ). The figure is nonetheless defensible: independent load-balancer benchmarks show HAProxy sustaining 50,000+ req/s with sub-millisecond latency on a modest 4-core VPS, and Envoy/NGINX are within ~10–15% of each other on raw HTTP/1.1 proxying — so 100K QPS on a modern multi-core gateway box is the right ballpark. Actual throughput still falls with plugin-chain length, payload size, and TLS cipher choice, which is exactly why it is a budget rather than a guarantee.

3. Capacity Estimation

Suppose the gateway fronts a moderately large public API: 10 million daily active users (DAU), each making an average of 50 API calls per day, with a peak factor of 5× over diurnal average.

3.1 Average and Peak QPS

total daily requests   = 10,000,000 DAU × 50 calls
                       = 5 × 10^8 requests/day

seconds per day        = 86,400

average QPS            = 5 × 10^8 / 86,400
                       ≈ 5,787 QPS

peak QPS (5× average)  ≈ 29,000 QPS

A single Kong/Envoy instance handles this comfortably (well below 100K QPS), so the dominant question is not “can one instance keep up” but rather “how many instances do we need for redundancy and zone diversity.” Three instances per availability zone, three zones — nine instances minimum, each running far below capacity for headroom.

3.2 Storage and Memory Per Instance

In-memory route table: 10,000 routes × 500 bytes/route metadata = 5 MB. Negligible.

Token introspection cache: if 1 million unique tokens see traffic per hour, and each cache entry is ~300 bytes (token hash + scopes + expiry), that’s 300 MB per instance — small. Cache hit rate determines whether we hammer the OAuth introspection endpoint or not; aim for > 99% cache hit on tokens (most tokens recur within their lifetime).

Rate-limit counters: per (key, route, window) tuple. With 100,000 active API keys × 50 routes × 3 window granularities = 15 million counters at peak. At 50 bytes each (key hash + count + window-start) = 750 MB — too much for in-memory per instance, hence the Redis-backed counter pattern (§8.3).

3.3 Bandwidth

average request body   ≈ 1 KB
average response body  ≈ 5 KB
peak bandwidth in      = 29,000 × 1 KB ≈ 29 MB/s ≈ 232 Mbps
peak bandwidth out     = 29,000 × 5 KB ≈ 145 MB/s ≈ 1.16 Gbps

Gigabit-class throughput per gateway tier — well within a single 10 Gbps network interface card.

4. API Design

The gateway itself exposes two distinct surfaces: a data plane (the public API the gateway proxies) and a control plane (the admin API to configure the gateway).

4.1 Data Plane (the public API the gateway fronts)

Whatever the upstream services define. The gateway does not invent endpoints; it transparently exposes the upstreams:

GET /api/v1/users/123                  → user-service
POST /api/v1/orders                    → order-service
GET /api/v1/products?q=shoes           → catalog-service

The gateway transforms paths, injects auth headers, and forwards. Optionally it aggregates (“BFF endpoint”) — see §8.4.

4.2 Control Plane (administrative)

A representative Kong-style admin API:

POST /admin/routes
{
  "name": "users-route",
  "paths": ["/api/v1/users", "/api/v1/users/*"],
  "methods": ["GET", "POST", "PUT", "DELETE"],
  "service": "user-service",
  "strip_path": true,
  "preserve_host": false
}

POST /admin/services
{
  "name": "user-service",
  "url": "http://user-service.internal:8080",
  "connect_timeout": 1000,
  "read_timeout": 30000,
  "retries": 2
}

POST /admin/plugins
{
  "name": "rate-limiting",
  "route": "users-route",
  "config": { "minute": 1000, "policy": "redis" }
}

POST /admin/consumers
{ "username": "client-app-mobile", "custom_id": "abc-123" }

POST /admin/consumers/client-app-mobile/key-auth
{ "key": "<generated-api-key>" }

The control plane is typically declarative (YAML applied via GitOps) in modern deployments rather than imperative-PUT-by-PUT, but the underlying primitives are these.

5. Data Model

The gateway holds three classes of state.

5.1 Configuration State (route table, plugins, consumers)

-- conceptual schema; concrete stores vary (Postgres for Kong, etcd for some, in-memory for Envoy fed by xDS API)
 
CREATE TABLE routes (
  route_id     UUID PRIMARY KEY,
  paths        TEXT[],
  hosts        TEXT[],
  methods      TEXT[],
  service_id   UUID,
  strip_path   BOOLEAN,
  priority     INT
);
 
CREATE TABLE services (
  service_id   UUID PRIMARY KEY,
  name         TEXT,
  protocol     TEXT,        -- http, https, grpc, ws
  host         TEXT,
  port         INT,
  retries      INT,
  timeout_ms   INT
);
 
CREATE TABLE plugins (
  plugin_id    UUID PRIMARY KEY,
  name         TEXT,
  scope_type   TEXT,        -- 'route' | 'service' | 'consumer' | 'global'
  scope_id     UUID,
  config       JSONB,
  enabled      BOOLEAN
);
 
CREATE TABLE consumers (
  consumer_id  UUID PRIMARY KEY,
  username     TEXT UNIQUE,
  custom_id    TEXT
);
 
CREATE TABLE credentials (
  credential_id UUID PRIMARY KEY,
  consumer_id   UUID REFERENCES consumers(consumer_id),
  type          TEXT,       -- 'key-auth' | 'jwt' | 'oauth2' | 'mtls'
  data          JSONB
);

The gateway loads this configuration into an in-memory representation (a trie keyed by path prefix is canonical for fast routing — see §8.1) and serves requests from memory; the database is only consulted on config change.

5.2 Hot State (rate-limit counters, circuit-breaker state, cached upstream responses)

This state is not in the configuration store. It lives in:

• Redis (or memcached) for rate-limit counters that span instances — a per-key counter must be globally consistent across the gateway fleet to enforce the limit accurately.
• In-memory for circuit-breaker state — each instance maintains its own view of upstream health, on the theory that local observation is the right input for local routing decisions. (Some designs gossip this; most do not.)
• In-memory or Redis for response caches.

5.3 Observability Output (logs, metrics, traces)

Sent to external systems: logs to an aggregator (see Logging Aggregation System Design), metrics scraped by Prometheus, traces propagated via OpenTelemetry to Jaeger or Zipkin (see Distributed Tracing System Design).

6. High-Level Architecture

flowchart TB
    Clients[External Clients<br/>mobile / browser / partners]
    Clients -->|HTTPS| LB[L4 Load Balancer<br/>anycast or DNS-based]
    LB --> GW1[Gateway Instance 1<br/>nginx + Lua / Envoy]
    LB --> GW2[Gateway Instance 2]
    LB --> GW3[Gateway Instance 3]

    subgraph GWInternals[Inside one gateway instance]
        TLS[TLS termination]
        Router[Route matcher<br/>trie / prefix-tree]
        AuthN[Auth plugins<br/>JWT / OAuth / mTLS]
        RL[Rate-limit plugin]
        Cache[Response cache plugin]
        Transform[Request/response transformer]
        CB[Circuit breaker]
        Forward[Upstream connector<br/>connection pool]
    end

    GW1 -.uses.-> GWInternals
    GW2 -.uses.-> GWInternals

    Forward -->|HTTP/1.1, HTTP/2, gRPC| Upstream1[user-service]
    Forward --> Upstream2[order-service]
    Forward --> Upstream3[catalog-service]

    subgraph CtrlPlane[Control Plane]
        Admin[Admin API]
        ConfigStore[(Config Store<br/>Postgres / etcd)]
        Watcher[Config Watcher<br/>etcd watch / pg_notify]
    end

    Admin --> ConfigStore
    ConfigStore -.notify.-> Watcher
    Watcher -.push config.-> GW1
    Watcher -.push config.-> GW2
    Watcher -.push config.-> GW3

    subgraph DataPlanePlugins[Stateful Plugin Backends]
        Redis[(Redis<br/>rate-limit counters,<br/>token introspection cache)]
        OAuth[OAuth Authorization Server<br/>Auth0 / Keycloak / internal]
    end

    RL --> Redis
    AuthN --> Redis
    AuthN -.introspect on cache miss.-> OAuth

    subgraph Observability[Observability Sinks]
        Metrics[Prometheus]
        Logs[Log Aggregator]
        Traces[Jaeger / OTel collector]
    end

    GW1 --> Metrics
    GW1 --> Logs
    GW1 --> Traces

What this diagram shows. The gateway tier is horizontally scaled behind a Layer-4 (L4) load balancer (anycast or DNS-based — see Load Balancer System Design). Each gateway instance is itself a pipeline: TLS termination, then a route matcher selects which route applies, then a chain of plugins runs (authentication, rate limiting, optional caching, request transformation), and finally the request is forwarded over a kept-warm connection pool to the matched upstream service. The chain is symmetrical on the response: response transformer, response logging, then back to the client. The control plane is a separate concern: an admin API writes to a config store; a watcher (etcd’s watch primitive, or PostgreSQL’s LISTEN/NOTIFY, or an out-of-band gossip channel) pushes config diffs to running instances, which swap their in-memory routing tables atomically without dropping connections (the key trick: see §8.5). Stateful plugins (rate limiting, token introspection cache) talk to Redis so that the gateway tier can be scaled horizontally without each instance enforcing its own siloed rate limit. Every instance emits logs, metrics, and traces to the observability stack — this is non-negotiable, because debugging a misbehaving gateway without these is essentially impossible.

7. Request Flow

sequenceDiagram
    participant C as Client
    participant LB as Load Balancer
    participant GW as Gateway
    participant R as Redis
    participant OA as OAuth Server
    participant U as Upstream Service

    C->>LB: HTTPS GET /api/v1/users/123<br/>Authorization: Bearer <jwt>
    LB->>GW: forward (TCP, TLS still terminated at GW)
    GW->>GW: TLS handshake → plaintext request
    GW->>GW: route match: /api/v1/users/* → user-service
    GW->>GW: JWT plugin: verify signature with cached JWKS
    alt token invalid / expired
        GW-->>C: 401 Unauthorized
    end
    GW->>R: GET ratelimit:apikey:abc:minute
    R-->>GW: 847
    alt count >= limit
        GW-->>C: 429 Too Many Requests<br/>Retry-After: 13
    else
        GW->>R: INCR ratelimit:apikey:abc:minute (TTL on first set)
    end
    GW->>R: GET cache:GET:/api/v1/users/123
    alt cache hit
        R-->>GW: cached body
        GW-->>C: 200 OK (body, X-Cache: HIT)
    else cache miss
        GW->>U: GET /users/123<br/>(rewritten path, injected X-User-Id header)
        U-->>GW: 200 OK { user data }
        GW->>R: SET cache:... (TTL=60s)
        GW->>GW: response transformer: strip internal headers
        GW-->>C: 200 OK { user data }, X-Cache: MISS
    end
    GW->>GW: emit log + metric + trace span (async, non-blocking)

Two latency-relevant observations. First, the JWT verification step is local — JWTs are signed, and the gateway holds a cached copy of the issuer’s JSON Web Key Set (JWKS, the public keys); verification is a CPU-bound RSA or Ed25519 signature check on the order of 100 microseconds, no network hop. This is why JWT-based auth scales to 100K QPS on commodity hardware; the alternative (token introspection over the network for every request) is 10–100× slower without the introspection cache. Second, the cache lookup and rate-limit-counter increment are both O(1) Redis round-trips, each ~0.5 ms on a same-zone Redis. The total gateway overhead is dominated by these network calls — pipelining or co-locating Redis with each gateway instance is a real optimization, not theoretical.

8. Deep Dive

8.1 Routing — the Trie at the Heart of the Gateway

Every incoming request must, in microseconds, be matched against thousands of route patterns. The naive approach — iterate over every route and test each pattern — is O(N) per request and falls over at hundreds of routes. The standard structure is a path trie (prefix tree), where each node represents one path segment and the final node holds the matched route’s metadata:

                    [root]
                   /      \
              "api"        "internal"
             /                  \
           "v1"                "v2"
          /    \                  \
      "users"  "orders"          "users"
       /          \                  \
    [route U1]  [route O1]        [route U2]

A request GET /api/v1/users/123 walks root → api → v1 → users (4 hops) and finds route U1; the trailing /123 is matched against U1’s path-parameter pattern. Lookup is O(depth of path), independent of N. With path parameters (/users/:id) and wildcards (/files/*) the trie has fall-back edges and the matcher prefers the most-specific match.

For host-based routing (api.example.com vs dev.example.com), the trie is keyed first on host then on path; in practice this is one trie per host, indexed by a hash on the Host header. Header-based and method-based routing add filter predicates at the leaf — these are usually evaluated linearly because there are few distinct sets of header constraints per route.

Spring Cloud Gateway uses a slightly different mechanism (a list of Route objects each with a chain of RoutePredicate and GatewayFilter objects, evaluated in declared priority order); Envoy uses an internal data structure called a RouteConfig with a similar trie underneath. Kong uses nginx’s location blocks compiled at config-reload time, plus a Lua-side router for dynamic routes.

8.2 Authentication Plugins

8.2.1 API Key Auth

Simplest. The client sends an opaque key in a header (X-API-Key: ...) or query string; the gateway looks up the key in its consumer table and resolves the consumer ID, which is then attached to the request as an injected header for upstream services. The lookup is over a hash table that loads from the config store at startup and updates via the config-watcher channel; per-request cost is a hash-map probe.

8.2.2 JSON Web Token (JWT) Auth

The client sends Authorization: Bearer <jwt>. A JWT (RFC 7519) is three Base64URL-encoded parts joined by dots: header (algorithm + key ID), payload (claims like sub, iss, exp, aud, custom scopes), signature. The gateway:

1. Decodes header to extract kid (key ID).
2. Looks up the public key for kid in its cached JWKS (JSON Web Key Set, fetched from the issuer’s /.well-known/jwks.json endpoint on a refresh interval, typically every hour).
3. Verifies the signature using the algorithm declared in the header (RSA, ECDSA, EdDSA).
4. Validates exp (expiry — must be in the future), iat (issued-at — must be in the past), nbf (not-before, optional), aud (audience — must include this gateway), iss (issuer — must be a trusted issuer).
5. Extracts authorization-relevant claims (scopes, roles) and uses them downstream.

The verification is local (CPU) and fast. The only network hop is the periodic JWKS refresh, which is rate-limited and cached. This is why JWT is ubiquitous in API gateways: stateless authentication scales without a backend.

JWT algorithm pitfall (verified against RFC 8725)

Pinning to specific algorithms is mandatory. Two classic attack classes motivate this. (1) The alg: none attack: some libraries accepted tokens declaring alg: none and skipped signature verification entirely. (2) Algorithm confusion (RS256 ↔ HS256): a verifier that lets the token’s own alg header select the verification path can be tricked into using an RSA public key as an HMAC secret, letting an attacker forge a valid signature. RFC 8725 (JWT Best Current Practices) addresses both directly: §3.1 requires that libraries “enable the caller to specify a supported set of algorithms” and “MUST NOT use any other algorithms,” and that “each key MUST be used with exactly one algorithm,” checked at verification time; §3.2 says libraries “SHOULD NOT consume JWTs using ‘none’ unless explicitly requested” (RFC 8725 §3.1–3.2). Operationally: the gateway must enforce a strict allowed-algorithm allowlist, never trust the alg header to choose between symmetric and asymmetric paths, and bind each key to exactly one algorithm.

8.2.3 OAuth 2.0 Token Introspection

Opaque (non-JWT) OAuth access tokens cannot be validated locally — the gateway must call the authorization server’s introspection endpoint (RFC 7662):

POST /introspect
Authorization: Basic <gateway-credentials>
token=<opaque-access-token>
→
{
  "active": true,
  "scope": "read:users write:users",
  "client_id": "mobile-app",
  "exp": 1730000000,
  "sub": "user-456"
}

Without caching, every API request triggers an introspection HTTP call — adding 5–20 ms per request and creating a hard dependency on the authorization server. Production gateways cache the introspection result keyed by the token (or token hash), with TTL set to either the response’s own exp claim or a configured maximum, whichever is sooner. Cache hit rates above 99% are typical because tokens are reused across many requests within their lifetime. The cache lives in Redis so that scaling out the gateway tier doesn’t cause cache-miss storms when sticky-session affinity isn’t perfect.

8.2.4 Mutual TLS (mTLS)

The client presents a client certificate during the TLS handshake; the gateway verifies it against a trusted certificate authority (CA) bundle, optionally checks revocation via Online Certificate Status Protocol (OCSP) or Certificate Revocation Lists (CRLs), and extracts the certificate’s subject as the consumer identity. Used heavily for service-to-service (B2B) APIs where API keys aren’t strong enough but full OAuth is too heavy. The latency cost is mostly absorbed in TLS handshake (one-time per connection); for kept-alive connections, mTLS is essentially free per request.

8.3 Rate Limiting

Three classic algorithms — see Token Bucket, Leaky Bucket, and Sliding Window Rate Limiter for the deep treatments. The gateway choice typically comes down to:

• Fixed window counter (simplest): per-(key, route, window-id) counter in Redis; INCR on each request; reject when ≥ limit. Cheap, but suffers the “double-burst at window edge” problem (a client can fire limit requests at 12:00:00.999 and another limit at 12:00:01.001 — effectively 2×limit over one second).
• Sliding window log (most accurate): per-key, store the timestamp of every request in a sorted set; on each request, drop entries older than the window and count what remains. Memory cost is O(limit) per key, which is large at scale.
• Sliding window counter (the production sweet spot): two adjacent fixed windows; the current count is count_current + count_previous × (1 - elapsed_in_current / window_duration). Approximation is within a few percent of the true sliding window with O(1) memory per key.
• Token bucket (smoothest): the bucket holds tokens; each request consumes one; tokens refill at a fixed rate. Allows brief bursts up to bucket capacity, then steady-state at the refill rate. The implementation in Redis uses a Lua script to atomically check-and-decrement to avoid race conditions.

The Redis-backed implementations all share one issue: the Redis hop adds 0.5–1 ms to every request. For ultra-low-latency tiers, gateways approximate distributed rate limiting via per-instance counters (each gateway enforces limit/N over a window, where N is the number of instances); this loses precision (a client can sometimes get 1.x × limit if traffic is uneven) but eliminates the network hop. The choice is workload-dependent.

Multi-dimensional rate limiting is the operationally interesting case: limit per-API-key (1000/min), AND per-IP (100/min), AND per-route (10000/min globally). Each dimension is a separate counter; the request is rejected if any counter exceeds. This means N counter increments per request, each a Redis round-trip — pipelining or MULTI/EXEC reduces N round-trips to one.

8.4 Request Aggregation — the Backend For Frontend (BFF) Pattern

A mobile-app screen often needs data from three or four backend services: user profile, notification count, recent orders, recommended products. The naive approach has the mobile app issue four parallel HTTPS requests, each with its own TLS handshake, auth, rate-limit check — ugly latency over a 4G link. The BFF pattern (Sam Newman, “Backends for Frontends,” 2015 — popularized in Building Microservices) puts a thin gateway-side layer in front of the mobile app that exposes one endpoint, fans out internally to the four services in parallel, and aggregates the results into one response:

GET /bff/mobile/home-screen
→ gateway aggregator:
    parallel:
      GET user-service/users/me
      GET notif-service/count
      GET order-service/recent
      GET catalog-service/recommend
    merge results → single JSON

Two design notes. First, the aggregation is per-client-class — there is typically a separate BFF for mobile, a separate one for web, possibly another for partner integrations — because each client class wants a differently-shaped composite. Second, error handling is interesting: if one of the four upstream calls fails, do you return a partial response (with a warnings field flagging the missing piece) or fail the whole composite? The defensible answer is partial responses, because mobile users vastly prefer “most of the screen rendered, plus a stale notif badge” to “infinite spinner.” This requires the client to be able to tolerate missing fields — a contract the BFF and the client maintain together.

GraphQL is an alternative to BFF at the protocol level: the client declares the shape of the data it wants in a single query, and the gateway (or a dedicated GraphQL gateway like Apollo Federation) fans out and stitches. The trade-off is increased complexity at the gateway tier in exchange for one fewer client-side roundtrip to add a new field.

8.5 Hot Configuration Reload

When the operator pushes a new route or rate-limit rule, the gateway must adopt it without dropping in-flight connections or restarting. The standard pattern:

1. Watch. Each gateway instance watches the config store (etcd watch, Consul watch, PostgreSQL LISTEN/NOTIFY, or polling). On change, the watcher fires a callback.
2. Build. The callback synthesizes a new in-memory routing table and plugin chain from the updated config — entirely in a separate object, not mutating the live one.
3. Atomic swap. A single pointer write switches current_routing_table from old to new. New requests begin matching against the new table; in-flight requests continue with whatever they had bound to.
4. Drain. The old table is held referenced by in-flight requests; once all references drop (Go’s runtime garbage collector or Lua’s reference counting handles this implicitly), the old table is freed.

Envoy’s xDS (extension Discovery Service) protocol is a well-known canonical implementation: the control plane streams config updates over gRPC; Envoy applies them via the same atomic-swap approach. xDS is now the de facto industry standard for gateway/sidecar config delivery — the same protocol Istio uses to configure its data plane (see Service Mesh System Design).

The subtle bug class is when one of the plugins holds non-routing state across requests (a counter, a connection pool, a bloom filter) — naive swapping discards that state. Production gateways track state lifetimes explicitly: state tied to a route version migrates with the route; state tied to a service ID survives version changes.

8.6 Circuit Breakers and Retries

Outbound calls to upstreams need protection. The circuit breaker pattern (Michael Nygard, Release It!, 2018) wraps each upstream behind a state machine with three states:

• Closed. Requests pass through. Failures are counted in a sliding window.
• Open. Failure rate exceeds threshold (e.g., 50% over the last 20 requests). All requests fail immediately with 503 without contacting the upstream — protects the upstream from being hammered while it recovers, and shaves 100ms+ off the user-facing latency.
• Half-Open. After a cooldown (e.g., 30 seconds), one probe request is allowed through. Success → return to closed. Failure → back to open.

Per-upstream, per-instance state is the standard model. Some designs gossip circuit-breaker state across instances; most do not, because local observation is faster and almost always correct.

Retries layer on top: each retry is a fresh request through the breaker. The critical discipline is the retry budget: cap the total number of retries per second or per second per upstream so that a flaking backend doesn’t get hammered into total failure by retry amplification. AWS’s “AWS SDK retry behavior” docs and Google’s SRE book both stress this — uncoordinated retries are the most common cause of cascading failures. A retry budget of “max 10% of original requests can be retries” is typical.

8.7 Blue/Green and Canary Deployments

The gateway is the natural place to implement traffic-shifting deployments because it is the routing decision point. For a canary deploy of user-service v2:

upstream selection for /api/v1/users/*:
  - 95% → user-service-v1 pool
  - 5%  → user-service-v2 pool

selection by:
  - hash on X-Request-Id (deterministic — same user gets same version)
  - or random (stochastic — distributes more evenly)
  - or by header (X-Beta-User: true → always v2)

The percentage is a config knob; flipping from 5% → 50% → 100% is one config push (with the hot-reload mechanism above). This is much cleaner than implementing canary at the application layer (where each service has to coordinate its own version selection logic).

Blue/green is a degenerate case: 100% → blue or 100% → green, with a single atomic flip.

9. Scaling Strategy

The gateway tier scales horizontally — gateway instances are stateless with respect to user data (user-specific state lives in Redis or the database).

9.1 Stage 1 — Single Instance

Up to ~30K QPS on commodity hardware (Kong/NGINX-class). The configuration database is co-located. No high availability — a process crash is full outage.

9.2 Stage 2 — Multiple Instances Behind L4 Load Balancer

Three or more instances behind an L4 LB (Direct Server Return for throughput). Configuration database now external (Postgres or etcd). Failure of one instance: the LB drops it from the pool after the next health check (~5 seconds). Rate-limit counters move to Redis to maintain global accuracy.

9.3 Stage 3 — Multi-Zone Per-Region

Three instances per availability zone, three zones per region. The L4 LB is itself zone-aware. Health checks detect zone failures. Capacity headroom: the system survives losing one zone (about 33%) without degrading.

9.4 Stage 4 — Multi-Region Active-Active

A separate gateway tier per region, fronted by a global anycast DNS (AWS Route 53 latency-based routing, Cloudflare Load Balancer with geo-steering — see Domain Name System Design). Requests are routed to the nearest region. Per-region Redis for rate-limit counters. Cross-region, the rate limit is enforced approximately (each region enforces limit/N, so the global limit is preserved on average but bursty across regions).

For tighter cross-region consistency, the rate-limit counter must move to a globally-replicated store (Spanner-class, multi-region DynamoDB) — which trades latency for accuracy. Most production systems accept the per-region approximation.

9.5 Stage 5 — Edge Gateway (Cloudflare Workers, Fastly Compute@Edge)

The gateway logic runs on the CDN’s edge. Requests never traverse the public internet to reach the gateway tier; the gateway itself is replicated to hundreds of points of presence (POPs) globally. Latency drops to single-digit milliseconds for end users far from the origin. The trade-off: limited compute/memory per invocation, and the configuration model differs (Cloudflare Workers KV for state, durable-object pattern for stateful coordination).

10. Real-World Example

Netflix Zuul. Originally Zuul 1 (servlet-based, blocking I/O — Netflix Tech Blog, 2013), later rewritten as Zuul 2 (Netty-based, non-blocking — Netflix Tech Blog, 2018) to handle millions of persistent connections without one-thread-per-connection overhead. Zuul fronts the entire Netflix streaming product: every call from the client app goes through Zuul before reaching the microservice mesh. Zuul integrates with Eureka (Netflix’s service discovery), Hystrix (the original circuit breaker, now retired in favor of Resilience4j), and Atlas (metrics). The 2018 rewrite was driven specifically by connection-scaling pressure: blocking I/O capped each instance at ~thousands of connections, while smartphone clients keep connections open for hours. Zuul remains in active use as Netflix’s edge gateway as of 2024–2025 — the Netflix/zuul repository is still maintained (release 2.1.8), and Netflix engineering talks continue to describe Zuul fronting nearly all consumer-facing device traffic (InfoQ — Netflix’s Edge Gateway Using Zuul). It has not been publicly replaced.

Kong. Built on top of OpenResty (nginx + LuaJIT). Plugins are written in Lua and run in the nginx event loop — no thread-per-request overhead. Kong’s configuration plane uses PostgreSQL or Cassandra; the data plane caches configuration in shared memory between nginx workers. Kong’s “DB-less mode” uses YAML config files for stateless deployments (Kubernetes-friendly). (Kong Inc. documentation.)

Envoy as a Gateway. Envoy was originally written at Lyft (Matt Klein 2017 KubeCon talk) as a sidecar proxy for service-to-service communication, but it doubles as an edge gateway. Envoy’s xDS protocol (a streaming gRPC API for config delivery) is now the de facto control-plane standard. Istio uses Envoy as its data plane (see Service Mesh System Design); Envoy-as-gateway products like Ambassador, Contour, and Gloo wrap Envoy with friendlier configuration UX.

AWS API Gateway. A managed service. Two flavors: REST API (full feature set, classic) and HTTP API (subset, lower latency, and substantially cheaper — $1.00 vs $3.50 per million requests, i.e. roughly 70% cheaper, per AWS API Gateway pricing as of May 2026). Integrates natively with AWS Lambda for serverless backends (the gateway invokes a Lambda function as the “upstream”), with AWS IAM for auth, and with AWS WAF for DDoS protection. The internal architecture is not publicly documented but is presumed to be a managed Envoy-class proxy fleet. (AWS API Gateway docs.)

Cloudflare Workers as Gateway. Workers run V8 isolates at every Cloudflare POP (~300+ globally as of writing). A gateway implemented as a Worker serves authentication, routing, and even simple aggregation entirely at the edge — origin servers are only contacted for cache misses or write paths. Used by major API providers (Discord’s media endpoints, Shopify’s storefront APIs) for sub-50ms global latency. (Cloudflare Workers documentation.)

Uncertain

Verify: the internal architecture of AWS API Gateway, Apigee, and other managed services (the “presumed managed Envoy-class proxy fleet” claim above). Reason: these internals are not published; the inference is from observable behavior and public talks, not a primary disclosure. To resolve: a vendor architecture deep-dive. The documented surfaces — pricing (verified, §10), feature sets, and quotas — are firm; the under-the-hood proxy implementation is not. #uncertain

11. Tradeoffs

Design Choice	Option A	Option B	When A wins	When B wins
Authentication style	JWT (stateless)	OAuth opaque + introspection	High QPS, scaling tier, no real-time revocation needed	Real-time token revocation needed; willing to pay introspection cost
Rate-limit backend	Redis (centralized)	Per-instance counter	Need accurate global limits, low-latency Redis available	Sub-millisecond gateway overhead is non-negotiable, OK with 1.x× limit drift
Config store	Postgres (relational)	etcd (key-value, watch native)	Existing Postgres ops expertise, complex relational config	Cloud-native, want xDS-style streaming config, already running etcd
Routing strategy	Path trie	Linear scan with regex	Thousands of routes, microsecond routing	Few routes, complex pattern logic
Aggregation	BFF gateway	GraphQL gateway	Distinct, stable per-client UIs	Heterogeneous clients, frequent field churn, willing to invest in schema federation
Deployment model	Self-managed (Kong/Envoy)	Managed service (AWS API Gateway, Apigee)	Compliance / on-prem requirement, cost-sensitive at large scale	Operational simplicity, pay-per-request makes sense at moderate scale
Process model	Event-loop (NGINX, Envoy)	Threaded (older Zuul, Spring Cloud Gateway)	Many concurrent connections, mostly I/O-bound	Plugins need blocking I/O or significant CPU work per request
Edge vs Origin gateway	Edge (Cloudflare Workers, Fastly)	Origin gateway only	Global user base, latency-critical	Compliance forbids edge processing of customer data

12. Pitfalls

1. Treating the gateway as a place to put business logic. “We can transform the order’s totals here” is a slippery slope; soon the gateway is doing tax calculations and inventory reservations. The discipline: the gateway does cross-cutting concerns (auth, rate limit, routing, transformation that is purely structural). Domain logic stays in services.

2. Synchronous fanout that can stall. A BFF endpoint that makes five upstream calls in parallel is fine — until one upstream is slow. Without per-call timeouts (often set lower than the gateway’s overall timeout), the slowest call dominates. Always set per-call timeouts and decide partial-failure semantics explicitly.

3. Token introspection storm on cache eviction. When an introspection cache entry expires for a popular token, hundreds of concurrent requests miss simultaneously and stampede the authorization server. Use single-flight (one introspection in-flight per token, others wait for the result) or staggered TTLs with jitter to avoid synchronized expirations.

4. Rate limit accuracy under multi-instance traffic. Per-instance counters give each instance a slice of the limit; if one instance gets disproportionate traffic (sticky session or hash skew), it exhausts its slice while others have headroom. Either centralize counters in Redis or use a leaky-bucket distributed token sharing scheme.

5. Hot reload that drops connections. A naive “rebuild config, kill workers, start new workers” reload pattern dumps every in-flight connection. The atomic swap pattern (§8.5) avoids this; verify it with a load test that reloads config every 10 seconds and checks for zero connection resets.

6. TLS termination without forward secrecy. Terminating TLS at the gateway is standard, but if the connection from gateway to upstream is plaintext, an internal-network attacker can sniff. Use mTLS or at least one-way TLS on the internal hop. In Kubernetes, this is what the service mesh sidecar handles automatically.

7. Missing observability on the gateway tier itself. Logs and metrics for upstream services are valued; the gateway’s own metrics (request rate per route, error rate by status code, plugin latency, connection pool utilization) are often neglected — until the gateway is the cause of an incident and there is no data. Plan the gateway’s own observability as carefully as that of services behind it.

8. No per-route circuit breaker. A single misbehaving upstream taking 60-second timeouts can drain the gateway’s connection pool, causing every other request to queue. Per-upstream circuit breakers protect the gateway’s resources, not just the upstream.

9. Open redirect via path rewrite. A misconfigured path-rewrite rule (/redirect?url=... blindly forwarded) creates an open-redirect vulnerability. Validate transform rules at config-load time; never trust path components as upstream-host targets.

10. Cache poisoning via Vary header oversights. Cached responses keyed only on URL — without considering Authorization, Accept-Language, Accept-Encoding — can return one user’s response to another. Either disable caching for authenticated endpoints, or include all relevant headers in the cache key (the Vary semantics in HTTP Caching, RFC 9111).

11. Deploying gateway changes without canary. The gateway is the most-blast-radius component in the system. Canary the gateway itself (route 1% of traffic to the new gateway version, monitor error rate, ramp up) — do not deploy a new version directly to 100%.

12. Treating health checks as routing decisions only. A passing health check tells you the upstream’s /health endpoint responds; it does not tell you the upstream can serve real traffic. Synthetic-transaction health checks (a probe that exercises the actual code paths the upstream serves) are more reliable.

13. Common Interview Variants and Follow-Ups

• “Add WebSocket support.” WebSocket connections start as HTTP/1.1 with Upgrade: websocket; the gateway must (a) match the route and authenticate the upgrade request, then (b) keep the underlying TCP connection open and pipe bytes between client and upstream without further parsing. Plugin chain typically stops after auth — rate limiting and most transformations don’t apply to WebSocket frames. NGINX and Envoy both support this natively.

• “Add gRPC support.” gRPC is HTTP/2 underneath with protobuf framing. The gateway must speak HTTP/2 to the client and to the upstream; routing is by :authority and the gRPC method path; some plugins (auth, rate limiting) work the same as HTTP/1.1; transformations that parse the body need protobuf-aware logic.

• “Implement a global rate limit of 1M req/s across the whole tenant fleet.” Per-instance counters won’t work; Redis with sharded counters; possibly an external service (e.g., Doorman from Google) that hands out leases. Discuss the tradeoff between accuracy and latency.

• “Add a request-replay attack mitigation.” Combination of timestamps + nonces + signed-request scheme (AWS SigV4 style). The gateway tracks recently-seen nonces in a short-lived cache.

• “Support per-tenant custom plugins.” Tenants want to inject their own request transformers. Sandboxing becomes the dominant concern: WebAssembly (Wasm) plugins (Envoy’s Proxy-Wasm spec, Kong’s Wasmer plugin) provide isolation and resource limits; raw Lua does not.

• “Migrate from monolith to microservices using the gateway.” The Strangler Fig pattern (Martin Fowler): the gateway routes a slowly-growing set of routes to new microservices, leaving the rest to the legacy monolith. Over time the monolith is decommissioned.

• “What if a single upstream service has 1000 different versions running at once for different tenants?” Per-tenant routing rules; the route table is keyed by (path, tenant-id); the configuration size grows linearly with tenants. Now hot reload latency matters: the trie has 1000× as many nodes.

• “Add request deduplication.” Mobile clients on flaky networks retry aggressively; if both retries succeed, you end up with two duplicate orders. The gateway can enforce idempotency keys (the client sends Idempotency-Key: <uuid>; the gateway caches the response per key for some window and serves the cached response on retry).

14. Open Questions / Uncertain

Uncertain

Verify: the internal mechanism AWS API Gateway uses to enforce rate limits / throttling across its multi-tenant fleet. Reason: not publicly documented; the “sharded Redis-class counter with consistent hashing” guess is inference, not a primary fact. To resolve: an AWS architecture disclosure or re:Invent deep-dive talk. The behavior (token-bucket-style account-level and per-method throttling with burst + steady-state limits) is documented; the implementation is not. #uncertain

The earlier flag asking whether Netflix Zuul 2 has been replaced is now resolved: as of 2024–2025 Zuul is still Netflix’s edge gateway and is actively maintained (see §10) — the Netflix/zuul repo and ongoing engineering talks confirm it has not been publicly superseded. “Netflix uses Zuul” remains a safe statement.

Uncertain

Verify: the production performance of WebAssembly (Wasm) plugins versus native Lua plugins in gateways. Reason: benchmark- and workload-dependent — published numbers range from “within ~20% of native” to “~2× overhead.” To resolve: a controlled, same-hardware benchmark on the specific plugin workload in question; there is no single representative figure. #uncertain

15. See Also

• Major System Designs MOC — parent MOC
• SWE Interview Preparation MOC — grandparent MOC
• Service Mesh System Design — sibling; the in-cluster equivalent (sidecar proxy)
• Load Balancer System Design — sibling; sits in front of (or merges with) the gateway tier
• Domain Name System Design — sibling; the routing layer above the gateway
• Content Delivery Network System Design — sibling; edge layer that often hosts gateway logic now
• Token Bucket — rate-limit primitive
• Sliding Window Rate Limiter — rate-limit primitive
• Leaky Bucket — rate-limit primitive
• Distributed Tracing System Design — observability for request flows through the gateway
• Logging Aggregation System Design — gateway log destination
• Metrics and Monitoring System Design — gateway metrics destination
• Consistent Hashing — for sharding rate-limit counters and stateful plugin data
• Centralized Configuration System Design — config-plane backing store (etcd, Consul) for the gateway control plane