vault

Cilium

16 min read

Cilium

Cilium is the eBPF-based Container Network Interface plugin for Kubernetes — the system that replaces the traditional iptables/IPVS-based kube-proxy, NetworkPolicy enforcement, and (optionally) sidecar service mesh with eBPF programs attached to Linux kernel hooks (TC ingress/egress, XDP, socket-layer, cgroup). It originated at Isovalent (founded 2017 by Thomas Graf and Daniel Borkmann — both core Linux-kernel eBPF maintainers) and was donated to the Cloud Native Computing Foundation at the Incubating maturity level on October 13, 2021. After two years of community growth, due-diligence review, and a third-party security audit, Cilium graduated to CNCF Graduated maturity on October 11, 2023 (cncf.io) — joining Kubernetes, etcd, Prometheus, Envoy, and a small number of other CNCF graduated projects. Cilium is the default data plane on Google’s GKE Dataplane V2, the data plane behind Azure CNI Powered by Cilium on AKS (learn.microsoft.com), and the most commonly chosen Cilium-replaces-VPC-CNI option on EKS when teams want eBPF performance over native VPC IPs. The architectural distinguishing feature is identity-based security: every workload is a CiliumEndpoint resource assigned a numeric security identity derived from its Kubernetes labels; policies are evaluated against identities (which are stable across Pod restarts) rather than IPs (which churn), so a CiliumNetworkPolicy like “allow app=web to call app=api” Just Works at scale across tens of thousands of Pods without reprogramming iptables on every Pod restart. Cilium also ships Hubble (cilium/hubble) — a deep observability layer that emits per-flow logs, builds a service-dependency map, and exports golden-signal metrics — and offers a sidecarless service mesh (Cilium Service Mesh) that delivers mTLS, L7 policy, and Gateway-API ingress without the per-Pod Envoy sidecar that Istio classic requires. ClusterMesh (docs.cilium.io) extends Cilium’s identity model across multiple Kubernetes clusters, enabling cross-cluster Pod-to-Pod routing and service load-balancing without a federated control plane. This note covers the eBPF data path, kube-proxy replacement, the CiliumNetworkPolicy policy model (including L7 HTTP/Kafka/gRPC), Hubble, and the multi-cluster story.

Mental Model

flowchart TB
    subgraph N1["Node 1"]
        subgraph P1["Pod A (identity=2105 'app=web')"]
            APP1[application<br/>10.244.1.5]
        end
        VETH1[lxc1234 veth<br/>+ TC ingress/egress<br/>eBPF programs]
        SOCK1[socket-layer eBPF<br/>service LB at connect()]
        AGENT1[cilium-agent DaemonSet<br/>compiles policy → eBPF<br/>maintains BPF maps]
        BPFMAP1[(BPF hash maps:<br/>endpoint identities,<br/>service backends,<br/>policy rules)]
        HUBBLE1[Hubble server<br/>per-flow events<br/>via perf-buffer]
    end
    subgraph N2["Node 2"]
        subgraph P2["Pod B (identity=3142 'app=api')"]
            APP2[10.244.2.7]
        end
        VETH2[lxc5678<br/>+ TC programs]
        AGENT2[cilium-agent]
        HUBBLE2[Hubble]
    end
    KVSTORE[(etcd or<br/>kvstore-less mode<br/>via CRDs)]
    OPERATOR[cilium-operator<br/>cluster-wide tasks:<br/>identity allocation,<br/>IPAM, ClusterMesh]
    HUBBLE_RELAY[Hubble Relay<br/>aggregates per-node<br/>flow streams]
    ENVOY[Embedded Envoy<br/>per-node L7 proxy<br/>for HTTP/Kafka/gRPC policy]
    APP1 -- "connect() to 10.96.0.42:80" --> SOCK1
    SOCK1 -- "socket-LB rewrites<br/>to Pod B's IP<br/>(no DNAT later)" --> VETH1
    VETH1 -- "TC egress: check policy<br/>(identity 2105 → 3142 allowed?)" --> N2
    VETH1 -. "L7 redirect<br/>if HTTP policy" .-> ENVOY
    AGENT1 --> BPFMAP1
    AGENT1 -- "identity allocate" --> KVSTORE
    OPERATOR --> KVSTORE
    HUBBLE1 --> HUBBLE_RELAY
    HUBBLE2 --> HUBBLE_RELAY

What this diagram shows. The eBPF-driven life cycle of a single Service-via-Cilium connection. Pod A (identity=2105, label app=web) on Node 1 wants to call a ClusterIP Service whose backend happens to be Pod B (identity=3142, label app=api) on Node 2. When Pod A’s application calls connect(10.96.0.42:80), the socket-layer eBPF program intercepts the connect() syscall, looks up the ClusterIP in a BPF hash map, picks a backend (Pod B), and rewrites the destination address at the socket layer — before the kernel emits any packet. The packet is then born with destination 10.244.2.7:8080 directly. This is a fundamental departure from iptables-based kube-proxy (which DNATs after the packet is in the IP stack, requires conntrack tracking, and pays per-rule O(n) cost) and even from IPVS mode (which is hash-based but still operates at the L3 layer). Cilium’s socket-LB is zero-conntrack for east-west traffic. On egress from Pod A’s veth, the TC egress eBPF program evaluates policy: it reads Pod A’s identity (2105) and the destination’s identity (3142) from BPF maps, looks up the policy decision in the policy BPF map, and either forwards or drops. If a CiliumNetworkPolicy requires HTTP method/path matching, the program redirects the packet to the per-node Envoy proxy for L7 inspection — Envoy is embedded as a sidecarless control point and only sees flows that need L7 evaluation. The right-hand side shows the observability path: every flow event is written to a perf buffer; the per-node Hubble server reads the buffer and streams events to a cluster-wide Hubble Relay, which aggregates and exposes them via gRPC for Hubble UI and Hubble CLI. The kvstore (etcd by default, optional in “kvstore-less” CRD mode) holds the identity allocations and per-cluster shared state. The insight to extract: Cilium replaces three different Kubernetes pieces (kube-proxy, the iptables-based NetworkPolicy enforcer, the sidecar mesh) with a single eBPF data plane. The simplification is architectural, not just operational — fewer hop counts, less state to manage, fewer race conditions between the iptables and IPVS subsystems and conntrack and userspace proxies. The cost is depth of kernel knowledge required to debug it.

Mechanical Walk-through

eBPF: what attaches where

Cilium’s eBPF programs attach at five primary hooks (docs.cilium.io):

  1. Socket layer (cgroup/connect4, cgroup/connect6, cgroup/sendmsg4, etc.) — intercept connect(), sendmsg(), bind(). This is where socket-layer load balancing happens: rewriting the destination IP of a connect() call so the packet is born with the backend’s address, no later DNAT needed. Eliminates kube-proxy for ClusterIP Services on a single-node path.
  2. TC ingress/egress on every Pod veth — packet-level processing for cross-node traffic, identity tagging in the packet’s IP options (or in the VXLAN ID for tunneled mode), policy evaluation, and load balancing for traffic that didn’t hit the socket layer (e.g., external NodePort traffic).
  3. XDP (eXpress Data Path) on the physical interface — earliest possible hook in the receive path. Used for DDoS protection, high-performance NodePort LoadBalancer, and sub-microsecond packet drops.
  4. TC clsact on cilium’s own internal devices (cilium_host, cilium_net) — host-to-host bridging when running in tunnel mode (VXLAN/Geneve).
  5. Cgroup hooks (cgroup_skb, sock_ops) — bandwidth manager, TCP socket tracing for Hubble, kube-proxy-replacement at the socket layer.

These hooks aren’t kernel modules; they’re verified-and-loaded BPF programs that run in a kernel-level sandbox. They cannot panic the kernel (the BPF verifier rejects unsafe programs), cannot loop forever (instruction count is bounded), and run at near-C speed.

Kube-proxy replacement

Cilium can fully replace kube-proxy (docs.cilium.io kubeproxy-free). The replacement supports four modes:

  • Strict (kubeProxyReplacement: true) — Cilium handles all Service types (ClusterIP, NodePort, LoadBalancer, ExternalIPs, sessionAffinity). kube-proxy is unnecessary and should be uninstalled.
  • Partial (deprecated names: probe, partial) — Cilium handles a subset; kube-proxy handles the rest. Used during migration.
  • Disabled (kubeProxyReplacement: false) — Cilium is just a CNI; kube-proxy still runs.

Performance (dev.to / Hubble FOSDEM 2023): the eBPF replacement maintains O(1) service lookup (hash map) regardless of service count, vs iptables’ O(n) chain traversal that becomes a measurable bottleneck above a few thousand services. The socket-LB path eliminates conntrack for east-west traffic (every Service hit no longer creates a conntrack entry), which can save significant conntrack-table pressure on busy nodes.

Identity-based security

The defining policy concept in Cilium (docs.cilium.io security/policy):

  • Every endpoint (Pod, External, World) is assigned a numeric security identity by the cilium-operator. Identities are derived from the endpoint’s labels (app=web,env=prod → identity 2105; any endpoint with the same labels gets the same identity).
  • Identities are stable: a Pod restarting with the same labels keeps its identity even though its IP changes.
  • Policies are expressed in terms of identities: from: endpointSelector(app=web) to: endpointSelector(app=api) translates to “identity 2105 → identity 3142 allowed.”
  • The eBPF data path tags every packet (in the inner IP header for tunneled mode, or a Linux mark for native-routing) with the source identity, so the receiving node can evaluate the policy without re-resolving the source IP.

This is the fundamental architectural advantage over IP-based firewalls. In an iptables-based world, a Pod restart with a new IP requires every NetworkPolicy referencing that Pod to be re-evaluated and re-programmed. In Cilium’s identity world, the identity is unchanged, so policy evaluation continues without disruption. At ~10,000 Pods churning at typical microservice rates, this is the difference between “policies converge in seconds” and “policies are always behind reality.”

CiliumNetworkPolicy: L3 to L7

Cilium accepts three policy types:

  1. Stock K8s NetworkPolicy (networking.k8s.io/v1). L3/L4 only — same expressiveness as any conformant CNI.
  2. CiliumNetworkPolicy (cilium.io/v2) — namespaced. Adds:
    • L7 HTTP rules: match method, path, headers. GET /api/v1/users allowed; POST /api/v1/admin denied.
    • L7 Kafka rules: match topic, API key (e.g., allow produce on events.*, deny on payments.*).
    • L7 gRPC rules: match method.
    • DNS rules: egress toFQDNs: ["*.googleapis.com"] — restrict egress by domain, not IP.
    • ICMP rules: allow specific ICMP types/codes (useful for traceroute exceptions).
    • Cilium endpoint selectors including aux:identity: matchers.
  3. CiliumClusterwideNetworkPolicy — cluster-scoped, applies across all namespaces; can match Cilium-specific reserved:host, reserved:remote-node, reserved:world identities for node-level policy.

L7 rules are enforced by the embedded Envoy proxy (docs.cilium.io). For each Pod that has an L7 policy applied, eBPF redirects the matching flow into Envoy, which parses HTTP/Kafka/gRPC and applies the rule. Envoy isn’t a sidecar — there’s one Envoy per node, embedded in the cilium-agent pod, processing flows for all Pods on that node.

Hubble: observability

Hubble (github.com/cilium/hubble, docs.cilium.io observability/hubble) is Cilium’s observability layer. Architecture:

  • Hubble server — runs in each cilium-agent Pod. Reads flow events from a kernel perf buffer that eBPF programs write to. Exposes a gRPC API on a Unix socket.
  • Hubble Relay — cluster-wide aggregator that opens a watch against every node’s Hubble server and merges streams into a single API.
  • Hubble UI — web frontend for the relay; renders a real-time service-dependency map.
  • Hubble metrics exporter — exposes Prometheus metrics derived from the flow stream: HTTP req/sec, latency histograms, error rates — the four golden signals (archive.fosdem.org 2023) per service pair, without app-side instrumentation.

The eBPF programs do not need to log every packet — they sample, aggregate per-flow, and emit structured events (PolicyVerdict, Drop, TraceL4, TraceL7). The result is observability without code changes: no instrumentation, no sidecars, no service-mesh interception.

Cilium Service Mesh

Cilium offers a sidecarless service mesh (Cilium Service Mesh) — instead of injecting an Envoy sidecar per Pod (the classic Istio model), Cilium uses its per-node Envoy for L7 policy and offloads basic mesh functions (mTLS, L4/L7 load balancing, retries, observability) into eBPF or that node-Envoy. Reduces per-Pod CPU/memory overhead; tightens the coupling between mesh and CNI. The full L7 feature set is comparable to Istio for HTTP/gRPC use cases; some advanced features (specific telemetry, certain auth flows) still favor classic sidecar Istio. See Ambient Mesh for Istio’s analogous sidecarless approach.

ClusterMesh: multi-cluster

Cilium ClusterMesh (docs.cilium.io clustermesh) connects multiple Kubernetes clusters into a unified Cilium network:

  • Each cluster runs its own Cilium control plane (kvstore, agents, operator).
  • A shared etcd / kvstore is exposed across clusters; cilium-agents on each cluster watch other clusters’ etcd to learn remote endpoint identities and IPs.
  • Pod-to-Pod connectivity works across clusters as if they were one — no gateway hops, no overlay tunnels per service.
  • Service load-balancing across clusters: a Service in Cluster A can have service.cilium.io/affinity: "remote" and traffic load-balances to backends in Cluster B too.
  • Identity is federated: the same labels in different clusters can share a global identity, so policies work uniformly.

The default on GKE-to-GKE multi-cluster, multi-region setups using GKE Dataplane V2.

CNCF graduation: October 2023

Cilium was accepted into CNCF Incubating on October 13, 2021 and graduated on October 11, 2023 (cncf.io, cncf.io announcement). At graduation, Cilium had become the second most active CNCF project by commit volume (behind only Kubernetes itself). The graduation required a third-party security audit, demonstrated production adoption at scale (Google, Adobe, Bell Canada, Datadog, IKEA, Sky, Trip.com, others), and the standard CNCF governance milestones.

Configuration / API Surface

A representative CiliumNetworkPolicy with L7 rules

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-allow-from-web
  namespace: shop
spec:
  endpointSelector:                            # (1) which Pods this policy protects
    matchLabels:
      app: api
  ingress:
  - fromEndpoints:                             # (2) which Pods may call us
    - matchLabels:
        app: web
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:                                   # (3) L7 rules (only enforced
        http:                                  #     because toPorts has rules)
        - method: GET
          path: "/api/v1/products(/.*)?"       # (4) regex path matching
        - method: POST
          path: "/api/v1/cart"
          headers:                             # (5) header matching
          - "X-Tenant: shop"
  egress:                                      # (6) DNS-based egress
  - toFQDNs:
    - matchPattern: "*.googleapis.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
  1. endpointSelector matches Pods by labels. Without namespaceSelector, scope is the policy’s own namespace.
  2. fromEndpoints is the L3 identity selector. In the data plane, this becomes a (srcIdentity, dstIdentity) allow-list entry in a BPF hash map.
  3. rules under toPorts activates L7 enforcement. The traffic is redirected to Envoy via eBPF for parsing.
  4. HTTP path matching supports regex. Envoy does the actual matching.
  5. Header matching is exact-string. Useful for tenant-scoped APIs.
  6. DNS-based egress is a Cilium-specific feature. Cilium tracks DNS responses (via the eBPF kprobe on the DNS query/response path) and dynamically allows the resolved IPs. Replaces the brittle “allow this IP range” with “allow this domain.”

Kafka L7 policy

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: kafka-tenants
  namespace: shop
spec:
  endpointSelector:
    matchLabels:
      app: kafka
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: orders-svc
    toPorts:
    - ports:
      - port: "9092"
        protocol: TCP
      rules:
        kafka:
        - role: produce                        # produce-only — can't consume
          topic: "orders.*"
        - role: consume
          topic: "shipments.events"

Install (Helm + cilium CLI)

$ cilium install --version 1.18.0 \
    --set kubeProxyReplacement=true \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true \
    --set ipam.mode=kubernetes
# After install:
$ cilium status --wait
$ cilium hubble enable
$ cilium hubble ui                              # opens http://localhost:12000

The CLI is the recommended install path — handles operator / agent / Hubble / kvstore in one go.

Failure Modes

  1. Kernel too old. Cilium’s full feature set requires Linux ≥ 5.4 (some features need 5.10+). On a kernel <4.19, Cilium won’t start; on 4.19–5.4, some features (XDP load balancing, socket-LB for UDP) are unavailable. Diagnostic: cilium status flags unsupported features. Fix: upgrade the node OS.
  2. Conflicting kube-proxy. If kube-proxy is still running and kubeProxyReplacement=true, the two implementations program overlapping kernel rules with non-deterministic results. Diagnostic: kubectl get ds -n kube-system kube-proxy. Fix: delete kube-proxy DS when using strict KPR.
  3. Envoy panic on L7 policy. A malformed L7 rule (unparseable regex) can cause the embedded Envoy to crash-loop. Symptom: L7-policy-affected flows fail; cilium-agent logs show Envoy errors. Fix: validate L7 rules with cilium policy validate before applying.
  4. Identity exhaustion. Cilium’s identity space is 16-bit (~65k) by default. Very-large multi-tenant clusters with many label combinations can hit the ceiling. Diagnostic: cilium identity list | wc -l. Fix: tune label normalization, or upgrade Cilium versions that support 24-bit identities.
  5. ClusterMesh PSA misconfiguration. ClusterMesh requires Pods on different clusters to have non-overlapping CIDRs and a shared kvstore TLS trust. Symptom: cross-cluster traffic fails silently. Diagnostic: cilium clustermesh status. Fix: ensure Pod CIDRs are unique across clusters, certs are in sync.
  6. DNS-based egress race. A Pod resolves a domain that returns multiple IPs over a long TTL; eBPF only tracks IPs seen in the DNS response. If the application caches an IP that the DNS rule never observed, traffic to it is dropped. Diagnostic: Hubble shows drops with reason “policy denied” on a domain that should be allowed. Fix: tune DNS TTL, ensure DNS responses go through the cilium-tracked path (typically CoreDNS).
  7. Hubble Relay overload. At very high flow rates (hundreds of thousands of flows/sec), Hubble Relay aggregating from many nodes can OOM. Diagnostic: relay logs + memory metrics. Fix: increase Hubble buffer size on agents (drops oldest events), scale relay horizontally, or use sampling.
  8. Native routing vs tunnel-mode confusion. Cilium supports two encapsulation strategies: tunnel (VXLAN/Geneve, default) and native routing (assumes underlay routes Pod CIDRs). If the underlay is configured for native routing but Cilium thinks it’s tunnel (or vice versa), cross-node packets are dropped. Diagnostic: cilium status | grep "Routing". Fix: align tunnel-protocol and enable-routing-mode config with underlay reality.

Alternatives and When to Choose Them

  • Calico — the established, conservative competitor. Calico’s iptables data plane works on every Linux kernel; Cilium needs modern eBPF. Choose Calico for stability and BGP-fabric integration; choose Cilium for eBPF performance, L7 features, and observability.
  • Flannel — strictly less. Cilium can chain on top of Flannel (Flannel handles veth, Cilium adds policy + L7). Rarely the right composition vs just deploying Cilium directly.
  • Weave Net — historical; do not pick for new clusters.
  • AWS VPC CNI — VPC-native IPs. Cilium can run alongside VPC CNI in policy-only mode (similar to Calico’s pattern), or replace it entirely with Cilium’s own IPAM for higher density.
  • Azure CNI + Cilium — Azure’s “CNI Powered by Cilium” is the recommended AKS option; combines Azure’s control plane with Cilium’s eBPF data plane.
  • Istio (sidecar) — for mesh use cases; richer telemetry per request, sidecar isolation, broader policy support. Cilium Service Mesh is lighter but doesn’t yet cover every Istio scenario.
  • Linkerd — Rust-based, opinionated, lighter than Istio. Smaller feature set than Cilium for CNI+mesh combined but easier operationally.

Production Notes

  • CNCF graduation announcement (cncf.io) — milestone-level adoption: Bell Canada, Trip.com, Sky, Datadog, IKEA, Adobe, Google, Capital One, S&P Global cited as production users. Cilium was the second-most-active CNCF project by commits at graduation.
  • GKE Dataplane V2 (cloud.google.com, docs.cloud.google.com dataplane-v2) — Cilium is the upstream for GKE Dataplane V2. On new GKE clusters, Dataplane V2 is the default. GKE exposes a subset of Cilium features to managed-cluster operators (e.g., NetworkPolicy, Hubble flow logs export to Cloud Logging).
  • Azure CNI Powered by Cilium (learn.microsoft.com) — combines Azure’s VNet-native IPAM with Cilium’s eBPF data plane and policy engine. The recommended AKS option for new production clusters.
  • Isovalent’s commercial offering — Isovalent (acquired by Cisco in 2024) sells Isovalent Enterprise for Cilium, with additional features (Tetragon for runtime security, enterprise support, enhanced Hubble). The OSS Cilium is fully functional; commercial features are operational enhancements.
  • Tetragon (tetragon.io) — Cilium’s sibling project for runtime security observability and enforcement, using eBPF kprobes/tracepoints to detect and (optionally) enforce process-level events. CNCF Incubating. Pairs naturally with Cilium but is a separate install.
  • Adoption pace — Datadog’s 2024 Container Report and CNCF’s 2024 surveys both showed Cilium being the fastest-growing CNI; by 2026 it has displaced kube-proxy and iptables-based policy as the default in most managed-K8s offerings.
  • Operational caveat — debugging Cilium requires kernel knowledge that iptables-trained operators don’t have. bpftool prog show, cilium bpf policy get, cilium endpoint get, perf-buffer reading — different toolchain than iptables -L. Plan for the learning curve.

See Also

Graph View

Backlinks