Load Balancer System Design

A load balancer (LB) is a network component that distributes incoming requests across a pool of backend servers, masking the pool behind a single virtual address while continuously monitoring backend health, terminating Transport Layer Security (TLS) when desired, and gracefully routing traffic away from failing or draining instances. The LB is a fundamental scaling primitive — without it, horizontal scaling of stateless services is essentially impossible. The discipline cleaves into two universes: Layer 4 (L4) load balancers, which route based on the TCP/IP 5-tuple (source IP, source port, destination IP, destination port, protocol) without inspecting the payload, and Layer 7 (L7) load balancers, which terminate TCP and reach into HyperText Transfer Protocol (HTTP) or gRPC headers and bodies to make smarter routing decisions. The canonical industrial deployments cross both worlds: HAProxy (the venerable software L7 LB since 2001), NGINX (web server doubling as software L4/L7 LB since 2004), F5 BIG-IP (the dominant hardware appliance), AWS Elastic Load Balancer in three flavors (Classic, Application — L7, Network — L4), Google Maglev (Eisenbud et al. NSDI 2016 — software L4 LB serving Google’s edge), Facebook Katran (eBPF-based L4 LB, 2018), and Cloudflare Unimog (eBPF-based, 2020). Modern infrastructure increasingly uses anycast for global ingress (one IP, multiple POPs, BGP routes to nearest), Direct Server Return for outbound throughput, and consistent-hashing variants for connection affinity under backend churn. This note covers the L4/L7 distinction in detail, the algorithms used to pick a backend, Maglev’s lookup-table-based hashing innovation, eBPF as the next-generation data plane, and the pitfalls that bite in production.

1. Why This System Appears in Interviews

Load balancing is the single most-deployed distributed-systems concept in production: every public service, every internal microservice, every database cluster has a load balancer somewhere. Interviews probe it because (1) the L4 vs L7 distinction tests whether the candidate understands the network stack — many candidates can describe HTTP routing but stumble when asked about kernel-level packet routing or how an L4 LB handles a TCP connection it cannot read; (2) the algorithm selection (round-robin, least-connections, weighted, consistent-hash) reveals whether the candidate understands the failure modes of each — a wrong choice causes hot backends, dropped sessions, or amplified failures; (3) the direct-server-return / Maglev / eBPF progression tests modern systems literacy — can you trace why Google rewrote Maglev or why Facebook moved from IPVS to eBPF? (4) the operational concerns — health checks, connection draining, sticky sessions, certificate management — are where production incidents happen, and the interviewer wants to see operational maturity. Load balancing also dovetails with Consistent Hashing, Rendezvous Hashing, Domain Name System Design (DNS-based LB above the TCP layer), API Gateway System Design (often co-located with the L7 LB), and Content Delivery Network System Design (CDN POPs are anycast LBs at scale), making it a “hub” topic from which an interviewer can branch in any direction.

2. Requirements

2.1 Functional Requirements

1. Request distribution. Spread traffic across N healthy backend instances. Support multiple selection algorithms.
2. Health checking. Continuously verify each backend’s health; remove unhealthy backends from the active pool; reintroduce them on recovery.
3. L4 (TCP/UDP) load balancing. Route based on TCP 5-tuple. Do not parse the application payload.
4. L7 (HTTP/gRPC) load balancing. Terminate TCP, parse the HTTP request, route based on host header, path, headers, cookies, or method.
5. TLS termination. Decrypt incoming TLS at the LB, optionally re-encrypt to backends. Manage certificate lifecycle (renewal, key rotation).
6. Sticky sessions / session affinity. Pin a client to a specific backend for the duration of a session, via cookie injection (L7), source IP hashing (L4), or Consistent Hashing on a session key.
7. Weighted distribution. Send more traffic to higher-capacity backends; useful for canary deployments (e.g., 5% to v2, 95% to v1).
8. Connection draining (graceful shutdown). When a backend is removed, existing connections finish their work; only new connections are routed elsewhere.
9. Layer 4 vs Layer 7 protocol awareness. Support HTTP/1.1, HTTP/2 (multiplexed streams over one TCP connection), HTTP/3 / QUIC (over UDP), gRPC, WebSockets, raw TCP/UDP.
10. Anycast routing for global ingress. Same IP announced from multiple POPs; BGP routes clients to nearest.

2.2 Non-Functional Requirements

1. Latency overhead. L4 LB: < 100 μs added latency. L7 LB: < 1-3 ms (TLS termination + HTTP parse + route lookup dominates).
2. Throughput. Per LB instance, support millions of requests per second (RPS) at L4; hundreds of thousands of RPS at L7.
3. Connection scalability. Tens of millions of concurrent connections per L4 instance (long-lived mobile connections); hundreds of thousands per L7 instance (HTTP-keepalive connections).
4. Availability. ~100% via HA pairs, anycast, or both. The LB is on the critical path of every request — its uptime bounds the service’s uptime.
5. Failover time. < 1 second from backend failure detection to traffic redirected away. Faster is better; sub-second matters at high QPS.

These targets are grounded in published benchmarks. At L4, the Maglev paper reports forwarding at 10 Gbps line rate — 813 Kpps for 1500-byte packets, and ~10 Mpps for small packets — per machine (Eisenbud et al. 2016, §3.2). At L7, HAProxy’s own benchmark forwarded over 2 million HTTP requests/sec on a single AWS ARM instance and 1.03 M HTTP RPS on an 8-core/16-thread Intel Xeon W2145 (~128 K RPS/core, plaintext). TLS termination is far heavier: in HAProxy’s SSL benchmark, enabling TLS 1.2 (ECDHE-RSA-AES128-GCM-SHA256) dropped a ~430 K conn/sec plaintext run to ~68 K new TLS connections/sec, and at 96 cores HAProxy established ~200 K new SSL connections/sec (~7,800/core). The lesson: the “millions of RPS at L4” and “100K+ RPS/core at L7” figures hold for forwarding and keep-alive throughput, but new-TLS-handshake rate is roughly an order of magnitude lower and is what actually bounds an HTTPS-terminating L7 tier.

3. Capacity Estimation

A representative public-API tier with 10 million daily active users (DAU), each making 50 requests per day, peak factor 5×.

3.1 RPS at the LB

total daily requests   = 10^7 DAU × 50 = 5 × 10^8
average RPS            = 5 × 10^8 / 86,400 ≈ 5,787 RPS
peak RPS               ≈ 29,000 RPS

This is small relative to a single modern LB instance (Maglev/Katran handle millions; HAProxy handles 100K+ per core). The LB tier is sized for redundancy, not capacity: 3 instances per availability zone, 3 zones = 9 instances at 30K RPS each, well within their headroom.

3.2 Connection Counts

If clients hold persistent HTTP connections for ~5 minutes on average:

concurrent connections = peak RPS × keep-alive duration
                       = 29,000 × 300 sec
                       ≈ 8.7 million concurrent connections

A single Linux server can handle this with kernel tuning (net.core.somaxconn, fs.file-max, ephemeral port range, conntrack table size). Distribution across the 9 instances drops it to ~1M per instance — comfortable.

3.3 TLS CPU Cost

TLS termination is the heaviest L7 cost. Modern AES-GCM on a CPU with AES-NI does ~5 Gbps per core; ECDSA handshakes at ~5,000 per second per core; TLS 1.3 with session resumption is ~30% cheaper than 1.2.

peak handshakes/s     = 29,000 RPS × (1/keepalive_factor=0.05)
                      ≈ 1,450 handshakes/s
core seconds needed   = 1,450 / 5,000 ≈ 0.3 cores

Below 1 core for handshakes; the symmetric crypto for kept-alive connections is bandwidth-bound, around 200-500 Mbps per core. At ~2 Gbps peak (the bandwidth math from §3.4), that’s ~5 cores busy on AES — a 16-core LB instance has plenty of headroom.

3.4 Bandwidth

average response       = 10 KB
peak bandwidth in      = 29,000 × 1 KB ≈ 29 MB/s ≈ 232 Mbps
peak bandwidth out     = 29,000 × 10 KB ≈ 290 MB/s ≈ 2.3 Gbps

Within the 10 Gbps NIC capacity of a modern server.

4. API Design

4.1 Wire Protocol — There Is No “API”

The LB has no application-level API; it speaks the underlying network protocol. Clients connect to the LB’s virtual IP (VIP); the LB forwards to a backend. The “API” of the LB is the wire protocol it understands.

4.2 Configuration API (Control-Plane)

Each LB has a control-plane configuration. Three styles are common.

HAProxy configuration (file-based):

frontend api_frontend
    bind *:443 ssl crt /etc/ssl/example.com.pem alpn h2,http/1.1
    mode http
    option forwardfor
    default_backend api_backend

backend api_backend
    mode http
    balance leastconn
    option httpchk GET /healthz HTTP/1.1\r\nHost:\ api.example.com
    http-check expect status 200
    server backend1 10.0.1.10:8080 check weight 100
    server backend2 10.0.1.11:8080 check weight 100
    server backend3 10.0.1.12:8080 check weight 50  backup
    timeout connect 1s
    timeout server  30s

NGINX configuration (file-based, similar):

upstream api_backend {
    least_conn;
    server 10.0.1.10:8080 weight=2;
    server 10.0.1.11:8080 weight=2;
    server 10.0.1.12:8080 weight=1 backup;
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name api.example.com;
    ssl_certificate     /etc/ssl/example.com.pem;
    ssl_certificate_key /etc/ssl/example.com.key;

    location / {
        proxy_pass http://api_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Envoy configuration (xDS dynamic, JSON / protobuf): A streaming gRPC API delivers updates incrementally; the data plane never restarts.

AWS Elastic Load Balancer: API-driven (elbv2 operations) or via Terraform/CloudFormation. Configuration is high-level (target groups, listeners, rules) rather than raw routing tables.

5. Data Model

The LB holds three categories of state.

5.1 Configuration State

The route table, backend pool, health-check definitions, TLS certificates. Static or rarely-changing. Loaded from a file or a control-plane API; held in memory.

5.2 Per-Connection State (L7 LBs Especially)

For each open TCP connection from a client:

• Client address, established time
• Selected backend and the backend-side connection
• For HTTP/2: per-stream state
• For sticky sessions: the affinity binding

L4 LBs may be stateless (Maglev’s hash-based selection produces the same answer for the same 5-tuple, no state needed) or stateful (track each TCP connection in a connection-tracking table — the conntrack approach used by Linux IPVS, Linux Netfilter, and AWS Network Load Balancer for TCP). Stateless L4 is faster and simpler under pod churn but loses the in-flight connection on backend changes; stateful L4 preserves connections but the conntrack table is finite and grows with concurrent flows.

5.3 Health-Check State

Per-backend: last probe timestamp, last status, success/failure streak counters. Used to decide pool inclusion.

6. High-Level Architecture

flowchart TB
    Internet[Internet Clients]
    Internet -->|TCP/443| Anycast[Anycast VIP<br/>announced via BGP from multiple POPs]
    Anycast --> POP1[POP 1<br/>L4 LB tier]
    Anycast --> POP2[POP 2<br/>L4 LB tier]
    Anycast --> POP3[POP 3]

    subgraph POP[Inside one POP]
        L4LB1[L4 LB Instance 1<br/>Maglev / Katran / Unimog]
        L4LB2[L4 LB Instance 2]
        L4LB3[L4 LB Instance 3]
        ECMP[Router with ECMP<br/>Equal-Cost Multi-Path]
        ECMP --> L4LB1
        ECMP --> L4LB2
        ECMP --> L4LB3

        subgraph L7Tier[L7 LB Tier in POP]
            L7LB1[L7 LB / Reverse Proxy<br/>NGINX / Envoy / HAProxy]
            L7LB2[L7 LB Instance 2]
        end

        L4LB1 --> L7LB1
        L4LB1 --> L7LB2

        Backend1[Backend Service A<br/>pods]
        Backend2[Backend Service B<br/>pods]
        L7LB1 --> Backend1
        L7LB1 --> Backend2
    end

    POP1 -.contains.-> POP

    subgraph CtrlPlane[Control Plane]
        Health[Health Checker]
        Config[Config Store / xDS]
        Cert[Certificate Manager<br/>Let's Encrypt / internal CA]
    end

    Health --> L4LB1
    Health --> L7LB1
    Config --> L4LB1
    Config --> L7LB1
    Cert --> L7LB1

What this diagram shows. Modern global load balancing is multi-tier. At the outermost layer, anycast announces the same Virtual IP (VIP) from multiple POPs (Points of Presence) globally; the BGP routing protocol routes each client to the topologically-nearest POP. Inside a POP, Equal-Cost Multi-Path (ECMP) routing on the gateway router spreads traffic across multiple L4 LB instances — each gets roughly an equal share by 5-tuple hashing. The L4 LBs do connection-level work (TCP termination handling, or pure forwarding in stateless mode) and pass to L7 LBs, which terminate TLS, parse HTTP, and route to specific backend services. The L7 LB is the layer where service-aware routing happens (path-based, host-based) — this is also where the API Gateway sits in many architectures, because the gateway and the L7 LB do overlapping work. Backend services receive cleanly-shaped requests over plaintext HTTP (often re-encrypted via mTLS in a service mesh). The control plane is separate: a health-check service continuously probes backends and updates the L4/L7 LBs’ active pools; a config store (file-based for HAProxy/NGINX, or xDS for Envoy) delivers configuration; a certificate manager (Let’s Encrypt’s certbot, or an internal CA) handles TLS material rotation. The diagram shows three POPs but a major service has 30-300; each POP runs several LB instances for redundancy.

7. Request Flow

7.1 L7 HTTPS Request End-to-End

sequenceDiagram
    participant C as Client
    participant DNS as DNS
    participant ANY as Anycast Router (BGP)
    participant L4 as L4 LB
    participant L7 as L7 LB
    participant B as Backend

    C->>DNS: A api.example.com?
    DNS-->>C: 198.51.100.7
    C->>ANY: TCP SYN to 198.51.100.7:443
    ANY->>ANY: BGP best-path → POP-NA
    ANY->>L4: forward (one of 3 instances by ECMP)
    L4->>L4: hash(5-tuple) → pick L7 instance
    L4->>L7: forward TCP segments
    L7-->>C: TCP SYN-ACK (TCP handshake completes)
    C->>L7: TLS ClientHello (SNI: api.example.com)
    L7->>L7: select cert by SNI; respond with cert
    Note over C,L7: TLS handshake completes (TLS 1.3, 1-RTT)
    C->>L7: HTTP/2 GET /v1/users/123<br/>Authorization: Bearer ...
    L7->>L7: route match by host+path<br/>backend pool: 8 healthy instances
    L7->>L7: load-balance: pick instance via least-connections
    L7->>B: forward GET /v1/users/123<br/>(HTTP/1.1 to backend, kept-alive)
    B-->>L7: 200 OK { user data }
    L7-->>C: 200 OK { user data }
    Note over L7: connection kept alive on both sides

The L7 LB does the bulk of the work: TLS handshake, HTTP parse, route match, backend selection, response forwarding. Each step adds tens to hundreds of microseconds; the total at p99 is typically 1-3 ms.

7.2 L4 LB with Direct Server Return (DSR)

sequenceDiagram
    participant C as Client
    participant LB as L4 LB
    participant B as Backend (with VIP on loopback)

    C->>LB: TCP SYN to VIP:443<br/>(MAC: LB's, IP-dst: VIP)
    LB->>LB: hash(5-tuple) → pick backend B<br/>rewrite L2 dst-MAC to B's
    LB->>B: forward (MAC: B's, IP-dst: still VIP)
    Note over B: VIP is on B's loopback — accepts.
    B-->>C: TCP SYN-ACK<br/>(IP-src: VIP, IP-dst: client)<br/>direct, bypassing LB
    Note over LB: Request flow only; response bypasses LB entirely.
    C->>LB: subsequent packets routed via LB
    LB->>B: forward
    B-->>C: response direct

DSR is a network-level trick: the LB rewrites the L2 destination MAC address but leaves the L3 destination IP as the VIP (which is configured on the backend’s loopback interface). The backend processes the packet normally and replies directly to the client — the response never traverses the LB. Since responses are typically much larger than requests (small request, large response), DSR roughly halves the LB’s bandwidth load. Used by Maglev, Katran, F5 BIG-IP, and many production LBs.

The trade-off: NAT (Network Address Translation) doesn’t work cleanly with DSR (since the client thinks it’s talking to the LB’s IP, but receives packets from the backend’s claim to that IP — only works because the VIP is on the loopback). DSR in its pure L2-rewrite form also requires the LB and backends to share an L2 broadcast domain; to forward across L3 networks the LB instead tunnels the packet to the backend (Maglev uses GRE, Katran uses IP-in-IP, Cloudflare’s Unimog uses GUE), with the backend decapsulating and still seeing the original VIP as the destination so it can reply directly to the client.

8. Deep Dive

8.1 L4 vs L7 — The Defining Distinction

L4 LB (Network/Transport layer): Routes based on the TCP/IP 5-tuple — source IP, source port, destination IP, destination port, protocol. Does not look inside the TCP segment. Properties:

• Very fast — packet inspection is microseconds.
• Protocol-agnostic — works for any TCP service (HTTP, SMTP, IMAP, custom binary protocols, raw TCP/UDP).
• Cannot make HTTP-aware routing decisions (path, host, cookies, headers).
• Sticky-session by source IP only.
• Cannot terminate TLS.

L7 LB (Application layer): Terminates TCP, parses the HTTP request (or gRPC, or whatever protocol), makes routing decisions based on the message content. Properties:

• Slower — HTTP parse, header inspection, possibly TLS handshake all add latency.
• HTTP-specific intelligence — host-based, path-based, header-based routing; cookie-based affinity; retries; rewrites.
• Can terminate TLS, freeing backends from TLS overhead.
• Sticky sessions via cookies (more reliable than source-IP, especially behind shared NAT).

In production, the two are typically stacked. The L4 LB at the front is fast and high-throughput, distributing TCP connections across L7 LB instances. The L7 LB does service-aware work. AWS exposes this directly: NLB (Network Load Balancer = L4) for TCP, ALB (Application Load Balancer = L7) for HTTP. Cloudflare’s Magic Transit is an L4 LB; their HTTP proxy is L7.

8.2 Backend Selection Algorithms

The choice of algorithm matters more than the choice of LB software.

8.2.1 Round-Robin

Cycle through backends in order: 1, 2, 3, 1, 2, 3, … Simple. Doesn’t account for backend capacity differences or current load. Fine for homogeneous backends with stateless workloads.

8.2.2 Weighted Round-Robin

Each backend has a weight; selection probability is proportional. Useful for backends with different capacities or for canary rollouts.

8.2.3 Least-Connections

Pick the backend with the fewest active connections. Good for backends with heterogeneous request durations (long-lived gRPC streams, slow database queries) — naturally balances by current load. Doesn’t work well with HTTP/2 (one connection multiplexes many requests, so connection count is decoupled from work).

8.2.4 Least-Time / Power-of-Two-Choices

Pick two backends at random; send to whichever has lower observed response time (or fewer connections). Surprisingly effective: the foundational analysis is Michael Mitzenmacher’s 1996 UC Berkeley PhD thesis “The Power of Two Choices in Randomized Load Balancing” (the widely-cited journal version is IEEE TPDS 2001), which proves that for n balls into n bins, choosing the lesser-loaded of two random bins makes the most-loaded bin hold only ln ln n / ln 2 + O(1) balls — exponentially better than the Θ(ln n / ln ln n) you get from a single random choice. Practically, it avoids the “thundering herd” failure of pure least-connections, where every LB simultaneously dog-piles the one backend that just freed up. NGINX implements it as the random two least_conn method, and many CDN edges use it.

8.2.5 IP Hash / Source-IP Hash

hash(client_ip) mod N selects the backend. Provides stickiness without cookies. Breaks under NAT (many users share one source IP) and when backends are added/removed (mod N shifts every assignment — the Consistent Hashing problem).

8.2.6 Consistent Hashing

Use Consistent Hashing on a session key (cookie, header, IP). Preserves stickiness across backend changes — adding/removing one backend only redistributes ~1/N of sessions. Used heavily for cache fleets where you want each cache key to hit the same backend (so the backend’s local cache is warm).

8.2.7 Rendezvous (HRW) Hashing

For each request, compute hash(key, backend) for every backend; pick the one with the highest score. See Rendezvous Hashing. Cleaner replication semantics than consistent hashing (top-K backends are easy: pick top-K by score) at O(N) cost per lookup. Cloudflare uses HRW for their load balancer. Trades larger per-request CPU for better load distribution.

8.3 Maglev — Google’s L4 LB Innovation

The 2016 NSDI paper by Eisenbud et al. introduced Maglev, a software L4 LB that handles Google’s massive ingress traffic. The key innovation: a lookup-table-based hashing scheme that combines O(1) lookup with consistent-hashing-like minimal disruption on backend changes.

The idea: build a fixed-size lookup table (say 65,537 slots) where each slot stores a backend ID. The 5-tuple of an incoming packet is hashed to a slot; the slot’s backend gets the packet. Building the table:

1. Each backend has a deterministic permutation of all slots.
2. Iterate: each backend, in turn, picks its highest-priority unfilled slot and claims it. Repeat until all slots are filled.

The math gives near-perfect uniformity and minimal disruption on backend changes. The paper quantifies it: with the default table size of 65537 (a prime, larger than the expected backend count), adding or removing a backend reshuffles only a small fraction of slots, and Maglev’s scheme needs far less overprovisioning to absorb the residual imbalance than the classic alternatives — at table size 65537, Karger-style consistent hashing and Rendezvous hashing require backends overprovisioned by 29.7 % and 49.5 % respectively, dropping to 10.3 % and 12.3 % only when the table grows to 655373 (Eisenbud et al. 2016, §5.3). Lookup is O(1) (just index into the table), unlike the O(log N) ring lookup of Consistent Hashing.

Maglev runs as a userspace process and forwards packets at line rate — typically 10 Gbps in Google’s clusters, which the paper notes is 813 Kpps for 1500-byte packets, and Maglev sustains line rate even with small packets (its forwarder can process on the order of 10 Mpps), adding only ~350 ns of latency per packet (§3.2, §3.4). It achieves this not with off-the-shelf DPDK but with its own kernel-bypass mechanism — a pre-allocated packet pool and busy-polling packet threads pinned to dedicated cores, talking to the NIC’s receive/transmit rings directly; the paper reports that introducing kernel bypass improved per-machine throughput “by more than a factor of five” over the standard kernel network stack (§4.2). On the wire, Maglev does not terminate TCP: it hashes the 5-tuple to a slot, then encapsulates the packet with Generic Routing Encapsulation (GRE) addressed to the chosen backend, and uses Direct Server Return so the (larger) response bypasses Maglev entirely (§2.3). It is deployed at every Google POP.

The same lookup-table-based hashing has been adopted elsewhere: Katran (Facebook) uses “an extended version of the Maglev hash” (FB engineering 2018), and Cilium’s eBPF data plane offers it via loadBalancer.algorithm=maglev for north–south service traffic (Cilium kube-proxy-replacement docs). The Maglev paper is the canonical reference.

8.4 eBPF and the Kernel-Path L4 LB (Katran, Cilium)

Traditional Linux L4 LBs (LVS/IPVS, Netfilter) run as kernel modules; userspace LBs (HAProxy, NGINX) terminate TCP in userspace at higher latency. eBPF (extended Berkeley Packet Filter) changes the game: programmable hooks in the kernel data path let you implement LB logic in safe, verified bytecode that runs inside the kernel — fast like a kernel module, but loadable on demand without recompilation.

Facebook Katran (open-sourced 2018) is an eBPF-based L4 LB. eBPF programs at the XDP (eXpress Data Path) layer hash the 5-tuple to a backend using “an extended version of the Maglev hash,” encapsulate the packet in IP-in-IP, and send it to the chosen backend — all without the packet ever entering the standard Linux network stack (FB engineering 2018). The blog runs Katran “alongside the backend servers in our PoPs.”

Uncertain uncertain

Verify: any specific Katran throughput number (e.g. “tens of millions of pps per server” or “~10× faster than IPVS”). Reason: an earlier draft of this note attributed those figures to the 2018 Facebook blog, but the blog itself gives no pps figures and no IPVS multiplier — it only states that performance is “measured as peak packets per second.” The “10× over IPVS” claim is folklore repeated by secondary write-ups, not a primary measurement. To resolve: a published, hardware-specified Katran-vs-IPVS benchmark. The order of magnitude (millions of pps on XDP) is plausible from XDP literature but the exact numbers are unverified.

Cloudflare Unimog (described publicly 2020-09-09) is Cloudflare’s edge L4 LB, also implemented in eBPF via XDP. Two things distinguish it. First, it runs on the same general-purpose servers that provide application services rather than on a dedicated LB tier — the blog says forwarding “costs less than 1 % of the processor utilization” on those servers. Second, it forwards using Generic UDP Encapsulation (GUE) (re-using GitHub’s glb-redirect component), and it does not use Maglev hashing — it hashes the connection 4-tuple to a key and looks the key up in a “forwarding table,” a deliberately simpler scheme than Maglev’s permutation table.

Cilium uses eBPF for both load balancing (a kube-proxy replacement that does service load balancing in eBPF at the socket and XDP/TC layers, with optional Maglev hashing and DSR for north–south traffic) and, more recently, a sidecar-less service-mesh data plane. This is the trend: as eBPF matures, network functions move from userspace and dedicated appliances back into the kernel — but as programmable, verifier-checked code rather than rigid kernel modules.

8.5 Health Checks — Active vs Passive

Active health checks: the LB periodically probes each backend (typically every 1-10 seconds) — TCP connect, HTTP GET to /healthz, etc. Backends failing N consecutive probes are removed from the pool. Backends passing M consecutive probes are reintroduced. Trade-offs:

• Probe interval too short: probe load competes with real traffic.
• Probe interval too long: failures detected late.
• “Synthetic” probes (e.g., GET /healthz returns OK) don’t always reflect actual health — a backend can return “OK” while its database connection pool is exhausted. Synthetic-transaction probes (probe an actual code path) are more reliable.

Passive health checks (outlier detection): observe real traffic; eject backends that fail too often. No probe load. Detects production failures, not synthetic ones. The tradeoff: a backend that’s truly broken sees real traffic fail before being ejected — a window of bad responses to real users. Both Envoy and Linkerd implement this as the default; HAProxy supports it via error-limit.

Most production deployments use both: active checks to detect total failures fast, passive observations to catch flaky-but-up backends.

8.6 Connection Draining (Graceful Shutdown)

When a backend is being removed (deployment, scale-down, manual operation), naive removal terminates in-flight connections — bad user experience. Connection draining:

1. Mark the backend as “draining” — new connections are not routed to it.
2. Existing connections continue normally.
3. After a configurable drain timeout (typically 30-300 seconds), terminate any remaining connections.

The drain timeout balances user experience (longer = more graceful) against deployment speed (longer = slower rollouts). HTTP/1.1 connections close naturally when the backend stops accepting new keep-alive requests; HTTP/2 connections are explicitly closed after streams complete via GOAWAY frames.

For long-lived workloads (WebSocket, gRPC streaming), draining requires application cooperation: the backend must close streams gracefully when signaled, not abandon them mid-message.

8.7 Sticky Sessions (Session Affinity)

Sometimes you must route a given user/session to the same backend repeatedly:

• Stateful backend (in-memory session store) — though most modern designs avoid this.
• Cache-friendly routing (the backend caches user-specific data; routing the user back to the same backend hits the cache).

L7 LBs implement stickiness via:

• Cookie injection. LB writes a cookie naming the chosen backend; subsequent requests with that cookie go to the same backend. The cleanest method.
• Cookie inspection. Application sets a cookie; LB hashes or reads it to pick the backend.

L4 LBs use source-IP hashing, which fails under NAT. For L4 stickiness in modern environments, use Consistent Hashing on a key the application provides (via a header at the L7 layer above) rather than source IP.

8.8 TLS Termination at the LB

Terminating TLS at the LB has several benefits:

• Backends don’t need their own certificates (centralized cert management).
• LB CPU handles TLS work (often on AES-NI-accelerated hardware).
• L7 LB can read HTTP and route intelligently.
• Inspection / WAF (Web Application Firewall) capabilities can examine plaintext traffic.

Trade-offs:

• LB ↔ backend traffic is plaintext unless re-encrypted. Many designs use mTLS on the internal hop (Service Mesh System Design).
• Cert management becomes the LB tier’s responsibility.
• TLS 1.3 with 0-RTT can leak retry information; configure carefully.

For some workloads, TLS pass-through (the LB does not terminate, just forwards encrypted bytes) is required — e.g., when the backend itself must validate the client’s identity via the certificate chain, or when end-to-end encryption is a regulatory requirement. Pass-through requires SNI (Server Name Indication) inspection at the TLS-handshake level to make routing decisions; the LB can read the SNI hostname (which is sent in plaintext) without terminating TLS.

8.9 HTTP/2 and gRPC at L7

HTTP/2 (RFC 7540) multiplexes many streams over one TCP connection. From the LB’s perspective:

• One TCP connection from the client carries many requests (streams). The LB must demultiplex and route each stream independently — the same connection might serve /api/v1/users/... (routes to user-service) and /api/v1/orders/... (routes to order-service).
• Backend connections also use HTTP/2 multiplexing; one backend connection can carry many streams to that backend.
• Connection-count-based load balancing breaks; per-stream metrics are needed.

gRPC sits on HTTP/2 with protobuf framing. Routing is by :authority and the gRPC method path (/Service.Method). Some L7 LBs (Envoy, NGINX with the gRPC module, HAProxy 2.x) speak gRPC natively; others terminate at HTTP/2 and treat gRPC as opaque body.

HTTP/3 (over QUIC, RFC 9000) shifts to UDP. Most LBs in production handle HTTP/3 by terminating QUIC at the L7 LB and re-establishing HTTP/2 to the backend; native QUIC-to-backend support is rolling out slowly.

8.10 Anycast for Global Ingress

The same VIP is announced from every POP via BGP. Each POP runs its own LB tier. Clients connect to whatever POP is “closest” by BGP path. Properties:

• Latency. Client reaches the nearest POP by network topology.
• Failover. If a POP fails, BGP withdraws the announcement; routing converges to the next-nearest POP within minutes (BGP convergence) or seconds (with BGP fast-failover features).
• Capacity. Aggregate capacity is the sum of all POPs.

Anycast is what powers Cloudflare, Fastly, Google’s edge, and AWS Global Accelerator. The LB at each POP is local; the global “load balancing” is done by BGP routing. See also Domain Name System Design §8.1 for anycast in DNS.

9. Scaling Strategy

9.1 Stage 1 — Single LB Instance

A small site with one HAProxy or NGINX. Single point of failure; works for hobby/dev.

9.2 Stage 2 — HA Pair (Active-Passive)

Two LB instances; one is active, one passive, with health checks between them. On active failure, the passive takes over (typically via VRRP — Virtual Router Redundancy Protocol — or keepalived). Failover is seconds.

9.3 Stage 3 — Active-Active with ECMP

Multiple LB instances behind a router doing Equal-Cost Multi-Path routing — each instance gets ~1/N of traffic by 5-tuple hashing. All instances active; failure of one shifts its share elsewhere via ECMP rebalancing.

9.4 Stage 4 — Multi-AZ Within a Region

Three LB instances per availability zone (AZ); three AZs. Survives AZ-level failures. Health checks span zones; failed zone is dropped from the pool within seconds.

9.5 Stage 5 — Multi-Region with Anycast or DNS-Based Routing

Each region has its own LB tier. Either anycast (BGP routes to nearest) or DNS-based (latency-based DNS returns regional VIP). Failover when a region degrades; clients reach a healthy region.

9.6 Stage 6 — Edge LB (Cloudflare / Fastly / AWS CloudFront)

The LB lives at hundreds of edge POPs globally. Latency drops to single-digit ms for most clients. The origin LB is now serving only cache misses and dynamic requests. Combines with Content Delivery Network System Design naturally.

10. Real-World Example

Google Maglev. Eisenbud et al. NSDI 2016. Software L4 LB serving Google’s edge. Uses lookup-table-based hashing (default table size 65537) for O(1) backend selection with minimal disruption. Userspace process with a custom kernel-bypass packet path; forwards at 10 Gbps line rate (813 Kpps for 1500-byte packets; ~10 Mpps for small packets) with ~350 ns added latency. Encapsulates with GRE and uses Direct Server Return. Deployed at every Google POP. The seminal paper introduced the lookup-table technique that subsequent eBPF LBs adopted.

Facebook Katran. Open-sourced 2018. eBPF-based L4 LB running at the XDP hook before packets enter the standard Linux network stack, using an extended Maglev hash and IP-in-IP encapsulation to forward to backends. Runs collocated with backend servers in Facebook’s PoPs. (The 2018 Facebook engineering blog states performance is measured in peak pps but publishes no specific pps figure or IPVS comparison — see the uncertainty flag in §8.4.)

Cloudflare Unimog. Publicly described 2020. eBPF/XDP-based L4 LB that runs on the application servers themselves (rather than dedicated LB hardware), eliminating an entire infrastructure tier and reportedly costing “less than 1 %” of those servers’ CPU. Forwards using Generic UDP Encapsulation (GUE) and a 4-tuple-hash-to-forwarding-table scheme (not Maglev’s permutation table). (Cloudflare blog 2020.)

AWS Elastic Load Balancing. Now four product types (the original three plus Gateway):

• Classic Load Balancer: legacy general-purpose L4 + L7 hybrid. AWS steers new deployments away from it toward ALB/NLB; it remains supported for existing users but is not the recommended choice for new work.
• Application Load Balancer (ALB): L7. Path-based routing, host-based routing, header-based routing, cookie stickiness, native HTTP/2 and WebSocket support. Auto-scaling; managed certs via AWS Certificate Manager (ACM).
• Network Load Balancer (NLB): L4. Static IPs (one per AZ), preserved source IP, very high throughput (millions of RPS), TCP and UDP. Used as the “first hop” in front of large fleets, often paired with ALB or Kubernetes ingress.
• Gateway Load Balancer (GWLB): operates at L3, used to insert third-party virtual appliances (firewalls, IDS/IPS, deep-packet-inspection) transparently into the traffic path via the GENEVE encapsulation protocol — a different use case from request distribution.

HAProxy. Open-source software L7 LB since 2001. The reference implementation for many features (rich health checks, advanced load-balancing algorithms, ACLs). Used at GitHub, Stack Overflow, Twitter, Reddit (per various engineering blog posts). HAProxy 2.x adds native HTTP/2 to backends, gRPC support, and Lua scripting.

NGINX. Originally a web server (2004), but now widely deployed as L7 LB and reverse proxy. Powers a large fraction of the web. NGINX Plus (commercial) adds active health checks and dynamic reconfiguration; open-source NGINX requires reload for config changes (though SIGHUP + graceful upgrade is essentially zero-downtime).

F5 BIG-IP. Hardware appliance with massive throughput (10s to 100s of Gbps per box) and a comprehensive feature set (WAF, SSL acceleration, HTTP rewrites). Dominant in enterprise data centers. Pricey; being slowly displaced by software LBs and cloud-managed services.

Linux IPVS (LVS — Linux Virtual Server). Kernel-mode L4 LB. Used as the data plane for Kubernetes services in IPVS mode (as opposed to iptables mode). Mature, fast, but conntrack-table-bound at scale. Increasingly replaced by eBPF-based alternatives.

11. Tradeoffs

Design Choice	Option A	Option B	When A wins	When B wins
Layer	L4 (5-tuple)	L7 (HTTP-aware)	Highest throughput, protocol-agnostic	Need path/host/cookie routing, TLS termination
Statefulness (L4)	Stateless (Maglev hash)	Stateful (conntrack)	Backend churn high, simplicity	Connection-preservation across LB restarts
Implementation	Software (HAProxy / NGINX)	Hardware (F5 / Citrix ADC)	Cloud-native, cost, flexibility	Enterprise compliance, regulated environments
Data path	Userspace (HAProxy)	Kernel (IPVS) / eBPF (Katran)	Operational simplicity, full TCP control	Maximum throughput, line-rate forwarding
TLS	Terminate at LB	Pass through	Centralized cert mgmt, L7 routing	End-to-end encryption requirement
Selection algorithm	Round-robin	Least-connections	Homogeneous backends, short requests	Heterogeneous request durations
Selection algorithm	Round-robin	Power-of-two-choices	Predictable	Want near-optimal balance with low coordination
Stickiness	None (stateless)	Cookie-based / consistent-hash	Stateless apps	Cache-friendly routing, stateful sessions
Health check	Active probe	Passive observation	Detect total failures fast	Detect flaky backends without probe load
Failover scope	Within region	Multi-region anycast	Single-region service	Global service, want regional outage tolerance
Direct Server Return	Yes	No	Outbound bandwidth-heavy (video, downloads)	NAT compatibility, simpler topology

12. Pitfalls

1. Source-IP hashing behind NAT. All users behind a corporate NAT share one IP; all are routed to the same backend; that backend gets hammered. Use cookie-based affinity at L7 instead.

2. Cache-stampede when an LB shifts backends. Removing a backend redistributes its sessions to others; if those sessions had warm caches, the new backends get cold-cache hits in droves. Use Consistent Hashing / Maglev to minimize redistribution.

3. Health check that lies. A /healthz returning 200 because “the process is alive” doesn’t reflect true readiness. Use synthetic transactions that exercise actual code paths.

4. Connection-tracking table exhaustion. L4 LBs with conntrack are bounded by table size (configurable but finite). Under DDoS or high concurrency, the table fills, new connections are dropped silently. Monitor conntrack utilization; consider stateless L4 (Maglev) for high-fan-in workloads.

5. TLS terminator certificate expiry. Forgotten cert renewal = sudden user-facing TLS errors. Automate via cert-manager / certbot; alert on certs expiring < 30 days.

6. No connection draining. Deploy or scale-down kills in-flight requests; users see 502s. Always configure drain timeout matching the longest request you serve.

7. Sticky sessions making graceful restart impossible. If a session is bound to backend X and X is removed, the session is reassigned — but if the application stored important state in X’s memory, that state is lost. Design backends to be stateless; or replicate session state.

8. L7 LB without rate limiting. A single noisy client can saturate the backend pool. Add per-IP / per-key rate limits at the LB (Token Bucket / Sliding Window Rate Limiter).

9. Forgetting Forwarded-For. L7 LBs that don’t add X-Forwarded-For (or RFC 7239 Forwarded) headers leave backends unable to log the real client IP. Backend security and analytics break.

10. Asymmetric routing without DSR. If the response path differs from the request path (e.g., the LB forwards in but the backend has a default route bypassing the LB on the way out), some firewalls will drop the response as “unsolicited.” Configure routing carefully.

11. WebSocket / gRPC streaming hitting idle timeouts. Long-lived connections idle longer than the LB’s idle_timeout; the LB closes the connection mid-stream. Increase the timeout for these workloads or use heartbeat keepalives.

12. HTTP/2 backend connection coalescing. With HTTP/2, the LB might multiplex many client streams onto one backend connection. If that connection slows down, all those streams slow down. Limit per-connection concurrency or open multiple connections per backend.

13. Anycast routing surprises. A client’s BGP path can change mid-session, sending follow-up packets to a different POP than the initial connection. The new POP doesn’t have the TCP/TLS state. Mitigate with consistent client-IP-based steering at the BGP level or session affinity tokens.

14. Cross-AZ traffic charges. In cloud environments, cross-AZ traffic has cost. A misconfigured LB sending 50% of traffic across AZs runs up bandwidth bills; configure zone affinity.

13. Common Interview Variants and Follow-Ups

• “Design a global load balancer for a service with 100M users.” Anycast at the IP level; per-region L7 LBs; CDN cache at the edge for static; multi-region active-active with cross-region replication. Failure detection via BGP and health probes.

• “How would you do canary deployments at the LB?” Weighted routing — 1% to v2, 99% to v1, monitor error rate. Increment if healthy. Automated by Argo Rollouts, Flagger, or hand-rolled tooling.

• “What’s the difference between AWS NLB and ALB?” NLB is L4 (TCP/UDP, lower latency, higher throughput, source-IP preserving). ALB is L7 (HTTP, path-based routing, WebSockets, AWS Lambda integration). Use NLB for high-throughput non-HTTP; ALB for HTTP-aware needs.

• “How would you implement zero-downtime deploys?” Connection draining + rolling deploy: mark instances draining one at a time, wait for drain timeout, deploy new version, add back as healthy. With multiple instances, capacity stays adequate throughout.

• “What is direct server return and when is it useful?” Section 8.3. Useful when responses are much larger than requests (video streaming, file downloads), to avoid the LB being the bandwidth bottleneck.

• “Trace a TCP connection from client to backend.” SYN to LB VIP → LB picks backend → LB forwards SYN (or rewrites and forwards depending on mode) → backend SYN-ACK → ack-handshake completes through LB or directly (DSR) → app data.

• “How do load balancers handle TCP fast-open and 0-RTT TLS?” TFO requires LB to forward the cookie consistently; some L4 LBs handle this transparently (Maglev, Katran). 0-RTT TLS requires careful configuration to avoid replay attacks; many LBs disable 0-RTT for non-idempotent requests.

• “Compare consistent hashing and rendezvous hashing for backend selection.” See Consistent Hashing and Rendezvous Hashing. Consistent hashing is O(log N) lookup, requires virtual nodes for balance. Rendezvous is O(N) lookup (compute hash for every backend) but cleaner top-K semantics and better default load distribution. For small N, rendezvous wins; for large N, consistent hashing is faster.

• “Why eBPF over kernel modules?” Kernel modules require recompilation, root privileges, and can crash the kernel. eBPF programs are verified at load time, bytecode-portable, hot-loadable, and safer (verifier proves they terminate and don’t access invalid memory).

• “How do load balancers interact with autoscaling?” When a backend scales up, the LB must learn about it (via service discovery — Kubernetes endpoints, Consul, etc.) and add it to the pool. When scaling down, drain → remove. Healthy scaling depends on the LB’s responsiveness to topology changes.

14. Open Questions / Uncertain

Uncertain uncertain

Whether eBPF-based L4 LBs (Cilium, Katran, Unimog) will fully displace conventional userspace LBs for L4 workloads in the next few years is a forward-looking judgement, not a verifiable fact. As of 2026, heavy production adoption is concentrated at hyperscalers (Google, Meta, Cloudflare) and in the Kubernetes/Cilium ecosystem; mid-market deployments still lean on managed cloud LBs or HAProxy/NGINX.

Uncertain uncertain

The performance characteristics of HTTP/3 / QUIC at the backend hop of L7 LBs are still evolving. The frontend (client-facing) side is now stable: NGINX shipped QUIC/HTTP/3 in 1.25.0 (2023), HAProxy added it from 2.5 (2021) with Enterprise 2.7 (2023) production-ready, and Envoy’s downstream HTTP/3 is production-ready (NGINX QUIC docs, Envoy HTTP/3 overview). What remains immature is upstream/backend HTTP/3 — Envoy explicitly labels its upstream HTTP/3 support “alpha … not tested at scale” — so most LBs still terminate QUIC at the edge and re-establish HTTP/2 to the backend. To resolve: track each LB’s upstream-QUIC GA status; this flag will age out as that matures.

Uncertain uncertain

Whether anycast or DNS-based geo-routing has “won” as the global ingress mechanism is a matter of architectural preference, not a settled fact. As of 2026, Cloudflare and Fastly are anycast-centric while AWS Route 53 still offers DNS-based latency/geo routing for many features; mixed deployments (DNS to steer to a region, anycast within it) are common.

Uncertain uncertain

Verify: the exact per-LB-instance vs total-fleet throughput tradeoff at multi-million-RPS scale. Reason: published numbers (Maglev’s 813 Kpps/10 Mpps, HAProxy’s 2 M RPS on an ARM instance) are real but hardware-, packet-size-, and workload-specific, and are best-case benchmarks from the vendors. To resolve: only a measurement on your own hardware and traffic mix is authoritative; treat published figures as upper bounds.

15. See Also

• Major System Designs MOC — parent MOC
• SWE Interview Preparation MOC — grandparent MOC
• Consistent Hashing — backend selection algorithm; minimal disruption on changes
• Rendezvous Hashing — alternative selection algorithm; cleaner top-K semantics
• API Gateway System Design — sibling; often co-located with the L7 LB
• Service Mesh System Design — sibling; the in-cluster equivalent (sidecar-as-LB)
• Domain Name System Design — the layer above LB; DNS-based load balancing predates anycast
• Content Delivery Network System Design — sibling; CDN edge POPs are anycast LBs
• Token Bucket — rate-limit primitive used at the LB
• Sliding Window Rate Limiter — alternative rate-limit primitive
• Distributed Lock Service System Design — sibling; consistent state for LB control plane
• Distributed Tracing System Design — observability for traffic through the LB
• Logging Aggregation System Design — LB log destination
• Metrics and Monitoring System Design — LB metrics destination