vault

Envoy Proxy

11 min read

Envoy Proxy

Envoy is a high-performance, C++-written Layer-7 proxy and “communication bus” originally built at Lyft (Matt Klein, 2015; open-sourced September 2016) and donated to the Cloud Native Computing Foundation, where it became one of the earliest CNCF Graduated projects in November 2018 — the third project ever to graduate, after Kubernetes and Prometheus (cncf.io). Envoy’s design goal was a single, universal data plane: the same proxy binary serving as an edge load balancer, a service-mesh sidecar, an internal L7 router, and a database/cache front proxy, so an organization makes one data-plane investment instead of five. It is the data plane of Istio (both sidecar and ambient modes), of many Gateway API implementations (Envoy Gateway, Contour, Emissary), of AWS App Mesh, of HashiCorp Consul’s connect proxy, and is embedded inside Cilium as the per-node L7 enforcement point. The architectural property that made Envoy universal is dynamic configuration via xDS — in mesh and gateway deployments Envoy holds no static config file; a control plane streams it listeners, routes, clusters, and endpoints at runtime over gRPC. This note covers Envoy’s configuration object model (listeners → filter chains → routes → clusters → endpoints), its L7 feature surface, its resiliency primitives (retries, timeouts, circuit breaking, outlier detection), and why it became the cloud-native default.

Mental Model

Envoy’s configuration is a tree of five object types, traversed once per request from the wire down to a destination host:

flowchart TB
    WIRE[Incoming connection<br/>on a bound socket]
    L[Listener<br/>binds an address:port]
    LF[Listener filters<br/>TLS inspector, http inspector,<br/>original_dst — pre-L4 inspection]
    FC[Filter chain<br/>selected by SNI / ALPN / SRC-IP match]
    NF[Network L3/L4 filters<br/>tcp_proxy, redis_proxy, ...]
    HCM[HTTP Connection Manager<br/>the key network filter:<br/>codec + HTTP filter chain]
    HF[HTTP filters<br/>router, ext_authz, ratelimit,<br/>fault, jwt_authn, cors, lua, wasm]
    RC[RouteConfiguration<br/>virtual hosts → routes:<br/>match path/header → cluster]
    CL[Cluster<br/>a named upstream service:<br/>LB policy, circuit breakers,<br/>outlier detection, conn pool]
    EP[Endpoints<br/>concrete host:port + health + weight + locality]

    WIRE --> L --> LF --> FC --> NF
    NF -.HTTP traffic.-> HCM --> HF --> RC --> CL --> EP

What this diagram shows and the insight to extract. A request enters a listener (an address:port Envoy binds). Listener filters run first — they peek at the raw bytes (TLS ClientHello SNI, HTTP method) before L4 processing to drive filter-chain selection. The matching filter chain is chosen by criteria such as SNI, ALPN, or source IP, and instantiates a stack of network (L3/L4) filters. For HTTP, the central network filter is the HTTP Connection Manager (HCM): it owns the protocol codec (HTTP/1, HTTP/2, HTTP/3) and runs a chain of HTTP filters ending in the router filter. The router consults a RouteConfiguration — virtual hosts and route rules that match on :path, :authority, headers — and resolves the request to a cluster (a named upstream service). The cluster owns load-balancing policy, connection pools, circuit breakers, and outlier detection; it dispatches to one of its endpoints (a concrete host:port). The insight: every Envoy capability is a filter or a cluster setting. Adding authn is adding an HTTP filter (jwt_authn, ext_authz); adding resiliency is a cluster’s circuit-breaker block. The object tree is also exactly the xDS resource hierarchy — LDS feeds listeners, RDS feeds routes, CDS feeds clusters, EDS feeds endpoints (see xDS Protocol).

Mechanical Walk-through

The threading model

Envoy is single-process, multi-threaded. A main thread handles xDS, stats flushing, and admin; a set of worker threads (one per core, typically) each run an event loop (libevent) and own a slice of connections via SO_REUSEPORT. Crucially, workers share almost nothing: each worker has its own copy of the (immutable) config snapshot, so a config update is a thread-local pointer swap with no locking on the data path. This is why Envoy can apply an xDS push without dropping connections — old connections finish on the old snapshot, new connections pick up the new one.

Protocols Envoy speaks

Envoy is a full L7 proxy, not just an L4 forwarder (envoyproxy.io arch overview):

  • HTTP/1.1, HTTP/2, HTTP/3 (QUIC) — and it transcodes between them: a client may speak HTTP/1.1 while Envoy uses HTTP/2 to the upstream. This protocol bridging is a major reason meshes use it — apps can stay on HTTP/1.1 while the mesh runs HTTP/2 internally.
  • gRPC — first-class: gRPC-Web bridging, gRPC-JSON transcoding, gRPC health checking and stats.
  • TCP and UDP proxying — tcp_proxy / udp_proxy network filters for non-HTTP protocols.
  • Protocol-aware filters — redis_proxy, mongo_proxy, dubbo_proxy, thrift_proxy, kafka — Envoy can L7-route Redis, Mongo, Kafka, etc.

Resiliency primitives

Envoy’s L7 resiliency is the feature set service meshes are built to deliver fleet-wide:

  • Retries — per-route retry_policy: which conditions retry (5xx, gateway-error, reset, connect-failure), num_retries, per_try_timeout, exponential backoff with jitter. Retries are also bounded by a circuit breaker (below).
  • Timeouts — per-route request timeout, idle timeout, per-try timeout; the x-envoy-upstream-rq-timeout-ms header lets callers override.
  • Circuit breaking — enforced per upstream cluster, per priority (envoyproxy.io circuit breaking): max_connections, max_pending_requests, max_requests (concurrent in-flight), max_retries (concurrent active retries — “retries for sporadic failures are allowed, but the overall retry volume cannot explode”), and max_connection_pools. Overflow increments counters like upstream_rq_pending_overflow and Envoy sets the x-envoy-overloaded header. This is a concurrency limiter, not a stateful open/closed breaker.
  • Outlier detection — the stateful “eject misbehaving host” mechanism: Envoy passively watches per-endpoint behaviour (consecutive 5xx, consecutive gateway errors, success-rate deviation from the cluster mean) and temporarily ejects an endpoint from the load-balancing set. Ejection duration grows with repeat offences (base_ejection_time × eject count); max_ejection_percent caps how much of the cluster can be ejected so a cluster-wide brownout doesn’t eject everything.
  • Load balancing — round-robin, least-request, ring-hash / Maglev (consistent hashing for session affinity), random; zone-aware routing; locality-weighted LB; subset load balancing (route to a labelled slice of endpoints — the mechanism behind Istio DestinationRule subsets).

Observability

Envoy is built to be observed. It emits thousands of statistics (counters, gauges, histograms) per cluster/listener/HTTP-route, exportable as Prometheus, statsd, or dog-statsd. Access logs are fully templated (gRPC, file, or stdout sinks) and record upstream cluster, response flags, durations. Distributed tracing is native — Envoy starts/continues spans and exports to Zipkin, Jaeger, Datadog, OpenTelemetry, propagating W3C Trace Context / B3 headers. Because Envoy sits on every hop, this gives uniform golden-signal telemetry without app instrumentation.

Configuration / API Surface

A minimal static bootstrap (the form used standalone; meshes replace most of this with xDS):

static_resources:
  listeners:
  - name: ingress_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }   # (1)
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager     # (2)
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          codec_type: AUTO                                       # (3) HTTP/1, /2, /3 auto-detect
          route_config:                                          # (4) inline routes (or rds: for dynamic)
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route:
                  cluster: user_service                          # (5) resolves to a cluster
                  timeout: 3s
                  retry_policy:
                    retry_on: "5xx,reset,connect-failure"        # (6) when to retry
                    num_retries: 2
                    per_try_timeout: 1s
          http_filters:
          - name: envoy.filters.http.router                      # (7) router MUST be last
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
 
  clusters:
  - name: user_service                                           # (8) the named upstream
    connect_timeout: 1s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST                                      # (9) LB algorithm
    circuit_breakers:                                             # (10) concurrency limits
      thresholds:
      - priority: DEFAULT
        max_connections: 1024
        max_pending_requests: 1024
        max_requests: 1024
        max_retries: 3
    outlier_detection:                                            # (11) eject bad endpoints
      consecutive_5xx: 5
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 50
    load_assignment:
      cluster_name: user_service
      endpoints:
      - lb_endpoints:
        - endpoint: { address: { socket_address: { address: user-svc, port_value: 80 } } }
  1. The listener binds 0.0.0.0:8080. In an Istio sidecar this address is 15006 (inbound) / 15001 (outbound) and is supplied by LDS, not written by hand.
  2. The HTTP Connection Manager is the network filter that makes this an HTTP listener — it owns the codec and the HTTP filter chain.
  3. codec_type: AUTO lets one listener serve HTTP/1.1 and HTTP/2 clients; HTTP/3 needs a UDP/QUIC listener.
  4. route_config is inline here; in a mesh it is rds: — dynamically fetched (see xDS Protocol).
  5. The route resolves the request to the cluster named user_service.
  6. retry_on enumerates retryable conditions; combined with num_retries and per_try_timeout.
  7. The router filter must be the terminal HTTP filter — it is what actually dispatches to the cluster. Filters before it (authz, rate limit, fault) run first.
  8. The cluster is the upstream service definition — the unit at which LB and resiliency are configured.
  9. LEAST_REQUEST sends each request to the endpoint with the fewest active requests.
  10. Circuit breakers cap concurrency — exceeding max_pending_requests fast-fails with 503 and x-envoy-overloaded.
  11. Outlier detection ejects an endpoint after 5 consecutive 5xx for base_ejection_time, never ejecting more than half the cluster.

Failure Modes

  1. Config rejected — NACK. An xDS push with an invalid resource is rejected by Envoy (it NACKs and keeps the last-good config). Symptom: control plane shows a NACK; Envoy keeps serving stale config. Diagnostic: /config_dump on the admin port, control-plane logs. This fail-safe is a feature — bad config can’t black-hole traffic.
  2. 503 UH / no healthy upstream. A cluster has zero healthy endpoints — EDS gave none, all health checks failed, or outlier detection ejected everything. Diagnostic: /clusters admin endpoint shows per-endpoint health and ejected flags.
  3. upstream_rq_pending_overflow. Circuit breaker max_pending_requests tripped — the upstream is slower than offered load. Symptom: fast 503s with x-envoy-overloaded. This is correct back-pressure, but a too-low limit causes spurious rejection.
  4. Memory growth under many clusters/endpoints. Each cluster and endpoint costs memory; a sidecar configured with the whole mesh (default Istio behaviour without Sidecar resource scoping) can use hundreds of MB. Fix: scope the config (Istio Sidecar CRD) so each proxy only learns the clusters it actually calls.
  5. Worker thread imbalance / hot connections. Long-lived HTTP/2 connections pin to one worker; an unlucky distribution overloads one core. Diagnostic: per-worker stats.
  6. HTTP/1.1 to HTTP/2 transcoding bugs. Header-case sensitivity, trailers, Connection-header handling — edge cases when bridging protocols. Diagnostic: access logs with response_flags.

Alternatives and When to Choose Them

  • NGINX / NGINX Plus — battle-tested edge proxy. Choose for static-config edge serving and as a web server. Envoy wins for dynamic (xDS) config, native gRPC/HTTP-3, and observability depth; NGINX’s dynamic reconfiguration is comparatively coarse.
  • HAProxy — superb L4/L7 load balancer with excellent performance. Choose for pure load balancing; Envoy wins for mesh integration and the filter/extension model.
  • Linkerd2-proxy (Rust micro-proxy) — see Linkerd. Far smaller memory footprint (~10–20 MB vs Envoy’s 50–300 MB) and tighter tail latency, but a deliberately minimal feature surface and not a general-purpose proxy. Choose it inside Linkerd; choose Envoy when you need the full L7 feature set or a unified edge+mesh data plane.
  • eBPF data plane (Cilium) — see Cilium Service Mesh. Moves L3/L4 into the kernel and removes per-pod proxies; still uses Envoy (per-node) for L7. Not an Envoy replacement — a re-partitioning of which layer does what.
  • Traefik — developer-friendly ingress with auto-discovery. Choose for simple Kubernetes ingress; Envoy/Envoy Gateway for mesh-grade L7 and Gateway API conformance.

Production Notes

  • Lyft — Envoy’s birthplace; Matt Klein’s 2017 “universal data plane” blog post (blog.envoyproxy.io) framed the thesis that one proxy should serve edge, mesh, and middle-proxy roles. Lyft runs tens of thousands of Envoy instances.
  • CNCF graduation — Envoy graduated November 2018, the third CNCF graduated project, signalling production maturity; it is now adopted across a majority of cloud-native enterprises (cncf.io).
  • The xDS standardization win — by 2026, xDS is a cross-proxy standard: gRPC libraries speak it directly (proxyless mesh), and non-Envoy proxies implement it. Envoy’s lasting contribution may be the API, not just the binary. See xDS Protocol.
  • Istio ambient — Envoy’s role shifted: in ambient mode (see Ambient Mesh) Envoy is the waypoint proxy (per-namespace, L7-only), while the per-node L4 layer is the Rust ztunnel, not Envoy. Envoy is no longer per-pod in modern Istio.
  • Operational caveat — Envoy’s configuration surface is enormous (thousands of fields, deeply nested protobufs). Almost nobody writes raw Envoy config in production; a control plane (istiod, Envoy Gateway, Gloo) generates it. Treat raw Envoy YAML as a debugging artefact, read via /config_dump.

See Also

Graph View

Backlinks