vault

kube-proxy

16 min read

kube-proxy

kube-proxy is the per-node daemon that implements the Kubernetes Service abstraction by programming kernel-level packet-routing rules so that any process connecting to a Service’s virtual IP (ClusterIP) lands on one of the Service’s backing Pods. It watches the API server for Service and EndpointSlice objects and translates them into iptables chains, IPVS virtual servers, or nftables maps — whichever mode is configured. The crucial design idea, and the most-misunderstood point about kube-proxy, is that kube-proxy is not actually a proxy in any of its modern modes: it does not carry packets, it only writes kernel rules and lets the Linux kernel’s netfilter / IPVS / nftables subsystems do the per-packet load balancing. The “userspace” mode that historically did proxy traffic in process was removed in Kubernetes 1.26. This note covers what kube-proxy does and the modern Linux backends — iptables (still the default), nftables (GA in 1.33 and the recommended Linux backend as of the v1.35 sneak peek), and ipvs (GA in 1.11 but deprecated in 1.35 and slated for removal in 1.36) — plus the Windows-only kernelspace backend and the eBPF-based replacements that obsolete kube-proxy entirely. The companion note kube-proxy Modes dissects each backend’s packet path; this note focuses on the daemon, its control loop, and its production behaviour.

Mental Model

flowchart LR
    APISERVER[(kube-apiserver)]
    subgraph "Node X (Linux)"
        KP[kube-proxy<br/>watches Services + EndpointSlices]
        subgraph "Linux kernel — packet path"
            NETFILTER[netfilter<br/>iptables / nftables rules<br/>OR IPVS hash table]
        end
        CLIENT[Client Pod]
        BACKEND1[Backend Pod A<br/>10.244.1.5]
        BACKEND2[Backend Pod B<br/>10.244.1.6]
        BACKEND3[Backend Pod C<br/>10.244.2.7]
    end
    APISERVER -- "watch Services, EndpointSlices" --> KP
    KP -- "iptables-restore / ipvsadm / nft -f<br/>(programs rules)" --> NETFILTER
    CLIENT -- "dial 10.96.0.42:80 (ClusterIP)" --> NETFILTER
    NETFILTER -- "DNAT to picked backend" --> BACKEND1
    NETFILTER -.-> BACKEND2
    NETFILTER -.-> BACKEND3

What this diagram shows. kube-proxy is on the control-plane path (it talks to the API server to learn which Services exist and which Pods back each one) but is off the data-plane path (the packet never touches kube-proxy’s process memory once the rules are installed). The kernel sees a connect() to 10.96.0.42:80 (a Service ClusterIP), matches a netfilter / IPVS rule that DNATs the destination to one of the backend Pod IPs, and forwards. The insight to extract: kube-proxy’s performance characteristics are the kernel’s performance characteristics with the rules kube-proxy installed — iptables is slow at large N because the kernel walks a chain linearly; IPVS is fast because it hashes; nftables is fast because the rule data structure was redesigned around sets and maps; eBPF is fastest because the rule itself becomes a JIT-compiled BPF program inserted at the socket level.

Mechanical Walk-through

kube-proxy is a Go binary, usually deployed as a DaemonSet in the kube-system namespace (kubectl get ds -n kube-system kube-proxy). On startup it reads its config (typically a ConfigMap mounted at /var/lib/kube-proxy/config.conf of type KubeProxyConfiguration), opens watches on the API server for Service and EndpointSlice (since K8s 1.21+ — historically Endpoints; EndpointSlice is the scalable successor that splits backends across multiple objects for Services with thousands of Pods), and runs a reconcile loop. Each time a Service or EndpointSlice changes, kube-proxy recomputes the desired set of kernel rules for that Service and writes them out via the chosen backend.

Mode 1: userspace (historical, removed). In K8s 1.0–1.1, kube-proxy actually listened on a node-local port for each Service, accepted connections, picked a backend via round-robin, and proxied bytes in process. This was slow (every byte went through user space twice) and limited. In 1.2 the default switched to iptables; the userspace mode was then deprecated in 1.23, stopped receiving feature work in 1.24, had its code deleted in 1.25, and was fully removed (fails to start) as of 1.26 (per the Kubernetes 1.26 removals blog and PR #112133). It is interview trivia only — and a common point of confusion: 1.23 was the deprecation, 1.26 the removal. The full timeline is dissected in kube-proxy Modes.

Mode 2: iptables — the default. Default on Linux since K8s 1.2 (Kubernetes Virtual IPs reference). For each Service kube-proxy creates a KUBE-SERVICES chain entry that matches the ClusterIP+port and jumps to a per-Service KUBE-SVC-<hash> chain. That chain contains one --probability rule per backend that DNATs to a KUBE-SEP-<hash> “service endpoint” chain, which in turn DNATs to the actual Pod IP+port. Backend selection is statistical random, not round-robin — netfilter has no per-connection state for “next backend in rotation”, so kube-proxy emits rules like -m statistic --mode random --probability 0.5 (for 2 backends), 0.333, 0.5 (for 3 backends, in cascade), and so on. Session affinity (spec.sessionAffinity: ClientIP) is implemented by netfilter’s recent module. Two tuning knobs matter: minSyncPeriod is the floor between rule rewrites under churn (the rule set is rewritten atomically with iptables-restore, so chatty Endpoints can otherwise melt the CPU), and syncPeriod is the ceiling between unconditional resyncs. The pathology of iptables mode is that rule evaluation is O(N) in the number of Services × backends per Service: at 1 000 Services with 10 backends each, every packet might walk thousands of rules before matching. In practice the kernel’s optimisations (RCU traversal, JIT-like rule caching) make this less bad than the bare complexity suggests, but the latency penalty becomes measurable above ~5 000 Services.

Mode 3: IPVS — scales to many Services. Promoted to GA in K8s 1.11. IPVS (IP Virtual Server) is a separate kernel module from netfilter that was designed for L4 load balancing: each Service becomes an IPVS virtual server and each backend an IPVS real server, stored in a hash table keyed on the Service VIP+port. Lookup is O(1), independent of Service count, so the per-packet cost stays flat at 10 000+ Services where iptables degrades. IPVS also offers richer scheduling algorithms (rr round-robin, lc least-connection, dh destination hash, sh source hash, wrr weighted round-robin) configurable via ipvsScheduler in the kube-proxy config. It still uses iptables for a small fixed set of auxiliary rules (masquerading, NodePort, packet marking) — IPVS only handles the VIP-to-backend translation. Switching to IPVS requires the ip_vs kernel modules loaded at boot and is enabled via mode: ipvs in the kube-proxy config. AWS EKS recommends IPVS for clusters above ~1 000 Services; below that, iptables overhead is negligible. As of Kubernetes 1.35 the IPVS backend is deprecated (KEP-5495, v1.35 sneak peek) and is intended for removal in 1.36 — the SIG Network maintainers cite the difficulty of keeping IPVS at feature parity with the other backends and the technical debt it carries. The official migration target for large Linux clusters is now nftables, not IPVS; new deployments should prefer nftables, and existing IPVS clusters should plan a migration ahead of the 1.36 removal.

Mode 4: nftables — the modern successor to iptables, now the recommended Linux backend. Introduced as alpha in K8s 1.29 (December 2023), promoted to beta in 1.31, and GA in K8s 1.33 — confirmed shipped, not merely targeted (nftables blog 2025-02-28; the v1.35 sneak peek states flatly that “for Linux nodes, the recommended kube-proxy mode is already nftables”). nftables is the Linux kernel’s modern replacement for iptables — same netfilter framework underneath, but the rule grammar is redesigned around sets and maps so that a Service-VIP-to-chain lookup is one hash operation rather than a linear chain walk, giving O(1) lookup versus iptables’ O(N). Performance is competitive with (and at very large Service counts surpasses) IPVS, while keeping the netfilter unification (one rule engine, not two) that simplifies the auxiliary rules. The one hard constraint: nftables mode requires a Linux kernel 5.13 or newer (nftables blog) and will not run on older distributions. iptables nevertheless remains the kube-proxy default even after nftables GA — explicitly stated in the blog: “even once nftables becomes GA, iptables will still be the default” — for backwards-compatibility and because not all CNIs that peek at iptables rules have migrated. So as of 2026 there is a deliberate gap between the default (iptables, for compatibility) and the recommendation (nftables, for scale): the project recommends nftables but has not flipped the default.

Windows: the kernelspace mode. Everything above is Linux. On Windows nodes kube-proxy runs in a single mode, kernelspace, which programs the Windows kernel’s networking (the Host Networking Service / Virtual Filtering Platform) to perform the VIP-to-Pod translation — there is no iptables/IPVS/nftables choice on Windows, and kernelspace is the only supported and default backend there (Virtual IPs reference). The userspace mode was removed on Windows in the same 1.26 cycle as on Linux. Mentioned here for completeness; the rest of this note is Linux-specific.

For each mode, kube-proxy also handles NodePort and LoadBalancer Services. NodePort exposes a Service on a port (default 30000–32767) on every node’s primary IP; the kube-proxy on each node installs a rule mapping <node-ip>:<nodeport> to the Service’s ClusterIP chain. LoadBalancer Services are NodePorts plus an external load balancer (provisioned by the cloud-controller-manager) pointing at every node’s NodePort. Pod-to-Pod traffic across nodes does not go through kube-proxy — it goes through the CNI data plane directly. kube-proxy only inserts itself in the path when a Service VIP is the destination.

A subtle but operationally important detail: kube-proxy by default masquerades Service traffic that comes from outside the cluster (NodePort/LB ingress), rewriting the source IP to the node’s IP via SNAT. This is necessary because the responding Pod has no route back to the original external client. The cost is that the backend Pod sees the node IP, not the real client IP. The workaround is Service.spec.externalTrafficPolicy: Local, which (a) only routes external traffic to backends on the same node and (b) preserves the source IP. The trade is unbalanced load (nodes with more backends get more traffic) but accurate client-IP logging.

”kube-proxy isn’t actually a proxy” — the load-bearing nuance

The name predates the implementation. In K8s 1.0–1.1, kube-proxy ran in userspace mode and was a proxy: every Service connection terminated in the kube-proxy process, which then opened a second connection to a chosen backend. The userspace mode was removed in K8s 1.23. In iptables, IPVS, and nftables modes — i.e. every mode supported in 2026 — kube-proxy does not see a single packet of Service traffic in steady state. It only writes kernel rules and stays out of the way. The name is now misleading. The eBPF-based replacements are even further from “proxy” semantics — they’re inline kernel programs at the socket layer.

The practical consequence: kube-proxy’s CPU usage scales with the rate of Service/Endpoint changes, not the rate of Service traffic. A cluster doing 1 M req/s through Services but with a stable topology will see kube-proxy idle. A cluster doing 100 req/s but rolling Deployments constantly will see kube-proxy spinning. Production tuning focuses on minSyncPeriod to batch rule rewrites and on EndpointSlice’s per-slice limit (100 endpoints per slice by default) so that a single Pod restart doesn’t dirty a massive rule set.

Configuration / API Surface

# kube-proxy ConfigMap (kube-system/kube-proxy/config.conf), mounted into the DaemonSet
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs                             # one of: "" (default = iptables), iptables, ipvs, nftables
clusterCIDR: 10.244.0.0/16             # the Pod CIDR; used to decide masquerade vs not
bindAddress: 0.0.0.0
healthzBindAddress: 0.0.0.0:10256
metricsBindAddress: 0.0.0.0:10249
 
# Mode-specific knobs
iptables:
  minSyncPeriod: 1s                    # don't rewrite rules more often than this
  syncPeriod: 30s                      # unconditional resync ceiling
  masqueradeAll: false                 # SNAT every Service packet (rarely needed)
  localhostNodePorts: true             # accept NodePort on 127.0.0.1 too
ipvs:
  scheduler: rr                        # rr | lc | dh | sh | wrr | nq | sed
  syncPeriod: 30s
  minSyncPeriod: 1s
  strictARP: true                      # needed if you also run MetalLB in L2 mode
nftables:
  syncPeriod: 30s
  minSyncPeriod: 1s
 
# Performance tuning
conntrackMin: 131072                   # set the kernel's max conntrack entries floor
conntrackMaxPerCore: 32768

To switch modes in a live cluster: update the ConfigMap, then kubectl rollout restart ds/kube-proxy -n kube-system. The mode switch is not disruptive in itself, but the brief interval where the new mode hasn’t yet finished programming rules while the old mode’s rules are being torn down can drop a fraction of Service connections; do this during a maintenance window for latency-sensitive workloads. On managed services the switch is exposed differently: AKS gates it behind a --kube-proxy-config flag, EKS lets you edit the upstream-shipped ConfigMap, GKE does not expose the mode at all (it’s iptables and you can replace kube-proxy with eBPF via Dataplane V2).

Failure Modes

Stale endpoints — “I deleted the Pod but traffic still goes there.” kube-proxy lags between EndpointSlice update and rule rewrite (minSyncPeriod). For a Pod that exits without terminationGracePeriodSeconds plus a working preStop hook + readiness-probe-driven removal, there’s a window of seconds where the Pod is dead but the rules still point at it. The fix is graceful termination plumbing: in the Pod, install a preStop hook that fails the readiness probe and sleeps for 2 × minSyncPeriod + RTT; the kubelet marks the Pod unready, EndpointSlice drops it, kube-proxy rewrites rules, then SIGTERM hits. The K8s docs call this out as the standard rolling-restart pattern; it’s the #1 source of “my deployment caused 500s” incidents.

iptables-restore failures. kube-proxy writes the entire rule set atomically with iptables-restore --no-flush --counters. If the rule set is malformed (almost always a CNI conflict — Calico’s Felix and kube-proxy both writing to overlapping chains), iptables-restore fails, kube-proxy logs Failed to execute iptables-restore: exit status 1 and rolls back to the previous state. Services keep working with stale rules until the conflict is resolved. Diagnose by iptables -t nat -L -n on the node and look for chains owned by something other than kube-proxy / Calico in the expected slots.

IPVS conntrack exhaustion. The kernel’s nf_conntrack table caps the number of tracked flows; at high connection rates with short-lived connections (think a Pod doing 50 k req/s to a Service) the table fills, new connections are dropped, and you see nf_conntrack: table full, dropping packet in dmesg. Tune conntrackMaxPerCore in the kube-proxy config or set nf_conntrack_max directly. IPVS does not avoid this — IPVS uses the standard conntrack table for stateful NAT.

Cilium-replacement + kube-proxy left running. If the operator enables Cilium’s kube-proxy replacement but forgets to delete the kube-proxy DaemonSet, both program rules and they fight. Symptom: intermittent Service unreachability. Fix: kubectl delete ds kube-proxy -n kube-system once Cilium’s kubeProxyReplacement: true is verified working. Cilium’s installer documents this and refuses to install with kube-proxy present unless explicitly told to coexist.

externalTrafficPolicy: Local causes uneven load. Setting Local to preserve client IP routes external traffic only to backends on the receiving node. If 3 of 10 nodes happen to have backends and 7 don’t, the cloud load balancer hashes 30 % of traffic to nodes with no local backend (those nodes return ICMP unreachable / drop), and the remaining 70 % goes through the 3 nodes that have backends — which now carry 7× their fair share. The cloud LB’s health checks should mark backend-less nodes unhealthy, but during rolling updates the health check lags. Mitigations: topology-aware backends, sufficient replicas per zone, externalTrafficPolicy: Cluster (which masquerades and loses client IP but spreads load).

Alternatives and When to Choose Them

Cilium kube-proxy replacement (eBPF). Since Cilium 1.6+, Cilium can replace kube-proxy entirely with eBPF programs attached at the socket layer (cgroup-bpf hooks) and at TC ingress on the host’s interfaces. Service-to-backend mapping is stored in an eBPF hash map, lookup is O(1), and for socket-level load balancing the translation happens before the packet is built — when the application calls connect(10.96.0.42, 80), the eBPF program rewrites the connect() arguments to the chosen backend’s IP directly, so the kernel never builds a “to-Service-VIP” packet at all. No DNAT, no conntrack involvement, no return-path SNAT. The performance win is substantial (Isovalent’s published benchmarks show 25–40 % CPU reduction on Service-heavy nodes), the observability win is larger (every Service decision is an eBPF tracepoint), and the feature win is real (eBPF can do XDP-rate LoadBalancer traffic without leaving the kernel). Enable via the Cilium Helm value kubeProxyReplacement: true; this requires kernel ≥ 4.19.57 (5.x recommended). See Cilium for the full story.

Calico eBPF data plane. Calico (since v3.11) also offers an eBPF mode that replaces kube-proxy. Less commonly deployed than Cilium’s; the trade-off is that Calico’s BGP-routed Pod networking is preserved while only the Service handling is BPF’d.

MetalLB / kube-vip + kube-proxy unchanged. For on-prem clusters that need LoadBalancer Services without a cloud LB, MetalLB or kube-vip assigns an external IP to a Service and announces it via BGP or ARP/NDP. kube-proxy still does the Service-VIP-to-backend mapping; MetalLB only handles getting traffic to the cluster. This is not really an alternative to kube-proxy — it’s a complement.

GKE Dataplane V2. Google’s managed Kubernetes ships Cilium under the hood as Dataplane V2 (a Cilium fork with deeper GCP integration). kube-proxy is absent. Selectable per-cluster at creation time.

Choosing (2026 guidance): stick with iptables under ~1 000 Services and a CNI that doesn’t conflict; for larger Linux clusters prefer nftables (GA since 1.33, kernel ≥ 5.13, the project’s stated recommendation) rather than IPVS — IPVS is deprecated as of 1.35 and will be removed in 1.36, so new IPVS deployments are a dead end and existing ones need a migration plan; deploy Cilium kube-proxy replacement if you’re already using Cilium as the CNI or if you have observability/policy needs that justify the operational change. Switching modes mid-life is operationally non-trivial and should be drilled in staging.

Production Notes

  • Spotify documented a kube-proxy-induced incident in their engineering blog series: a malformed Service spec caused iptables-restore to fail across the fleet; the cluster kept running on stale rules until the malformed Service was deleted, but new Services and Endpoint changes were not reflected. The post-mortem recommended a CI-time validator for Service objects.
  • AWS EKS at scale. EKS clusters above ~3 000 Services routinely switch to IPVS mode; the EKS Best Practices Guide (aws-eks-best-practices/networking/ipvs) covers the rollout. Above ~10 000 Services even IPVS’s rule churn becomes a concern and the recommendation moves to Cilium.
  • Datadog and Cloudflare have published kube-proxy elimination posts: both swapped to Cilium kube-proxy replacement and reported double-digit CPU reductions on dense nodes.
  • CIS Benchmark. The CIS Kubernetes Benchmark (1.8+) flags kube-proxy default permissions; a hardened deployment runs kube-proxy with --feature-gates=AllAlpha=false and a system-cluster-critical priority class so it survives node-pressure eviction.
  • Conntrack and long-lived connections. Long-lived gRPC connections through a kube-proxy iptables-mode Service do not rebalance when new backends are added — conntrack pins the flow to its original backend until the connection closes. This is a frequent surprise for engineers debugging “why are my new replicas not getting traffic.” The fix is server-side connection age limits (MAX_CONNECTION_AGE in gRPC) so clients periodically reconnect.

See Also

Graph View

Backlinks