vault

Hubble

13 min read

Hubble

Hubble is the observability layer of Cilium — it turns Cilium’s eBPF data plane into a source of deep, identity-aware network visibility with no application changes, no sidecars, and no code instrumentation. Because Cilium’s eBPF programs already sit on every packet’s path in the Linux kernel (see Cilium for the data-plane mechanics), Hubble gets network telemetry essentially for free: it reads the structured flow events those eBPF programs already emit. Hubble provides flow logs (a record of every L3/L4 connection — and, where L7 visibility is enabled, every HTTP/gRPC/Kafka/DNS request — between workloads), service-dependency maps (an automatically discovered graph of which service talks to which), golden-signal metrics (request rate, error rate, and latency per service pair), and network-policy verdict observability — the ability to answer the question that is otherwise miserable to debug: “which policy dropped this packet?” Critically, every Hubble flow is identity-aware: it is labelled with the Cilium security identity of the source and destination (derived from Kubernetes labels) plus full Kubernetes context — Pod name, namespace, labels, service name — not just an IP address. This note covers Hubble’s architecture (the per-node server, Hubble Relay, the UI and CLI), what it observes, why eBPF makes it cheap, and how it compares to sidecar-based and agent-based observability.

Mental Model

flowchart TB
    subgraph N1["Node 1"]
        EBPF1["Cilium eBPF programs<br/>(on every Pod veth, socket layer)"]
        PERF1[(kernel perf ring buffer<br/>structured flow events)]
        HS1["Hubble server<br/>(embedded in cilium-agent)<br/>gRPC API on Unix socket"]
        EBPF1 -- "PolicyVerdict, Drop,<br/>TraceL4, TraceL7 events" --> PERF1
        PERF1 --> HS1
    end
    subgraph N2["Node 2"]
        EBPF2["Cilium eBPF programs"]
        HS2["Hubble server<br/>(in cilium-agent)"]
        EBPF2 --> HS2
    end
    RELAY["Hubble Relay<br/>(cluster-wide aggregator —<br/>watches every node's server)"]
    UI["Hubble UI<br/>(service-dependency map,<br/>flow visualization)"]
    CLI["Hubble CLI<br/>(hubble observe —<br/>real-time flow stream)"]
    METRICS["Hubble metrics exporter<br/>(Prometheus: rate / errors / duration)"]
    HS1 --> RELAY
    HS2 --> RELAY
    RELAY --> UI
    RELAY --> CLI
    HS1 -. "per-node metrics" .-> METRICS
    HS2 -. .-> METRICS
    METRICS -. "scraped by" .-> PROM["Prometheus / Grafana"]

What this diagram shows. Hubble’s data originates where Cilium already is: the eBPF programs attached to every Pod’s virtual interface and to the socket layer (the Cilium note covers exactly where they attach). As those programs process packets, they already make decisions — was this allowed by policy? was it dropped? trace it — and they write structured flow events (PolicyVerdict, Drop, TraceL4, TraceL7) into a kernel perf ring buffer. The Hubble server, embedded inside each node’s cilium-agent Pod, reads that buffer and exposes the flow stream over a gRPC API on a Unix socket — per-node visibility. To get cluster-wide visibility, Hubble Relay opens a watch against every node’s Hubble server and merges the streams into one aggregated API. The Hubble UI consumes the Relay to render a live service-dependency map; the Hubble CLI (hubble observe) streams and filters flows from the same Relay; and the Hubble metrics exporter derives Prometheus golden-signal metrics from the flow stream. The insight to extract: Hubble adds no new data path — it reads the events Cilium’s eBPF programs already produce. That is why it costs almost nothing: there is no packet copy, no sampling proxy, no sidecar interception, no application SDK. The observability is a byproduct of the data plane.

Mechanical Walk-through

Why eBPF makes this cheap

Traditional network observability has three expensive shapes: a sidecar proxy per Pod (a service mesh — every packet detours through userspace Envoy, doubling hop count and adding CPU/memory per Pod); application instrumentation (every service imports a tracing SDK and emits spans — invasive, polyglot-unfriendly, easy to get wrong); or packet capture at the NIC (tcpdump-style — high overhead, no Kubernetes context, just IPs). Hubble has none of these costs because Cilium’s eBPF programs are already on the packet path doing the actual networking work (docs.cilium.io). They are already evaluating policy, already load-balancing, already tracing — so emitting a structured event about what they just did is nearly free: a write into a ring buffer the kernel and userspace share. There is no extra hop, no packet copy into userspace for inspection, no sidecar. The marginal cost of a flow event is a few instructions in an eBPF program that was going to run anyway. This is the entire economic argument for eBPF-based observability: the telemetry is a byproduct of the data plane, not a tax on it.

Flow logs — the core primitive

A flow is Hubble’s atomic record: a single L3/L4 connection (or, with L7 visibility on, a single L7 request) between two endpoints. Where a classic firewall log gives you src IP → dst IP : port, a Hubble flow record carries (oneuptime — Cilium flow logs):

  • Source and destination Kubernetes identity — Pod name, namespace, labels, and the Cilium security identity number — not just IPs. This is the single biggest difference from IP-based logging: a flow says frontend → checkout, by name, even though both Pods’ IPs churn on every restart.
  • Protocol detail — L4 (TCP/UDP, ports, flags) and, with L7 visibility, the HTTP method/path/status, gRPC method, Kafka topic/API-key, or DNS query/response.
  • The policy verdict — FORWARDED, DROPPED, ERROR — and which policy produced it.
  • Direction, traffic type, timestamps.

Flows can be streamed live, filtered, and — via the Hubble flow-log exporter — written to disk as JSON for ingestion into a log pipeline.

Service maps and golden signals

Because every flow names its source and destination service, Hubble can aggregate flows into a service-dependency graph with no configuration — it discovers the topology by watching traffic. The Hubble UI renders this as a live map at L3/L4 and, where enabled, L7.

From the same flow stream, the Hubble metrics exporter derives golden-signal metrics (archive.fosdem.org 2023 — Golden Signals with Cilium and Grafana): per service pair, the request rate, error rate, and duration/latency — the RED method (Rate, Errors, Duration) — exported as Prometheus metrics and dashboarded in Grafana. The point worth internalizing: these are normally the metrics you get only after instrumenting every service with a tracing/metrics SDK. Hubble produces them from kernel-level flow observation, with zero application code changes.

Network-policy verdict observability

This is Hubble’s killer operational feature. Debugging “service A cannot reach service B” with iptables-based NetworkPolicy is miserable — you stare at iptables chains and guess. Hubble makes it a single query: every flow carries its policy verdict, so hubble observe --verdict DROPPED shows you exactly which connections were denied, by which policy, between which identities. The “which policy dropped this packet” question — the single most common, most painful network-policy debugging task — becomes a filter on a stream. This is observability of enforcement, not just of traffic.

L7 visibility

By default Hubble observes L3/L4. When a Cilium L7 policy (or explicit L7 visibility annotation) applies to a workload, Cilium’s eBPF redirects the relevant flows through the per-node embedded Envoy, which parses the application protocol — and Hubble then sees L7 detail: HTTP method/path/status code, gRPC method, Kafka topic and API key, DNS queries and responses. L7 visibility is opt-in per workload because parsing has a (small) cost; L3/L4 visibility is always on.

Identity-aware, not IP-aware

Every Hubble flow is keyed on Cilium security identity — a stable numeric identity Cilium derives from a workload’s Kubernetes labels (see Cilium for the identity model). A Pod that restarts with the same labels keeps its identity even though its IP changes. So a Hubble flow log remains semantically stable across Pod churn: it says app=web → app=api consistently, where an IP-based log would show a meaningless, ever-changing pair of addresses. At microservice scale with constant Pod turnover, identity-awareness is the difference between flow logs you can reason about and noise.

Configuration / API Surface

Enabling Hubble and querying it:

# --- Enable Hubble (Relay + UI) on an existing Cilium install ---
$ cilium hubble enable --ui                         # (1) turns on Relay + UI
 
# --- Check status ---
$ cilium status | grep Hubble                       # (2) Hubble: OK, Relay: OK, UI: OK
 
# --- Open the service-dependency map in a browser ---
$ cilium hubble ui                                  # (3) port-forwards the UI
 
# --- Stream live flows from the whole cluster (via Relay) ---
$ hubble observe --follow                            # (4) real-time flow stream
 
# --- The killer policy-debug query: what is being DROPPED? ---
$ hubble observe --verdict DROPPED \
      --namespace shop --last 200                    # (5) every denied flow in 'shop'
 
# --- Identity-aware filtering: flows from one workload to another ---
$ hubble observe --from-label app=web \
      --to-label app=api --protocol http             # (6) by label/identity, not IP
 
# --- L7 HTTP detail (requires L7 visibility on the workload) ---
$ hubble observe --protocol http --http-status 500   # (7) every HTTP 500 in the cluster
  1. cilium hubble enable --ui installs Hubble Relay (the cluster-wide aggregator) and the UI. The per-node Hubble server is already embedded in every cilium-agent Pod — enabling Hubble just turns it on and adds the Relay.
  2. cilium status confirms the three pieces: the per-node server, the Relay, the UI.
  3. The UI renders the auto-discovered service-dependency map — no configuration; it is built from observed flows.
  4. hubble observe --follow is the live tail of every flow cluster-wide, served by the Relay.
  5. --verdict DROPPED is the network-policy debugging workhorse — it answers “what is being blocked” directly, replacing the iptables-chain guessing game.
  6. --from-label/--to-label filter by Kubernetes label (identity), not IP — stable across Pod restarts. This is the identity-aware advantage in CLI form.
  7. --http-status 500 filters on L7 detail — available when the workload has L7 visibility enabled, parsed by the embedded Envoy.

Hubble metrics are enabled via Cilium config (hubble.metrics.enabled) listing the metric families to export (dns, drop, tcp, flow, http, etc.) — Prometheus then scrapes the per-node Hubble metrics endpoint.

Failure Modes

  1. Hubble Relay overload at high flow rates. Relay aggregates flow streams from every node; at hundreds of thousands of flows/second across a large cluster it can OOM or fall behind. Diagnostic: Relay logs and memory metrics. Fix: increase the per-agent Hubble buffer (it drops oldest events when full — flow logs are best-effort, not guaranteed), scale Relay, or sample.
  2. Buffer overflow drops flows. Each node’s Hubble server reads from a fixed-size ring buffer; if the consumer falls behind, the oldest events are overwritten. Hubble flow logs are therefore best-effort observability, not an audit log — do not treat a missing flow as proof a connection did not happen. For guaranteed records, use the flow-log exporter to disk and accept its overhead.
  3. L7 detail missing. hubble observe --protocol http shows nothing because the workload has no L7 policy or L7-visibility annotation, so eBPF never redirects its traffic through Envoy for parsing. Fix: apply an L7 Cilium policy or the L7 visibility annotation to the workload.
  4. UI shows an incomplete service map. The map is built from observed flows — a service that has not communicated recently, or whose flows were buffer-dropped, may be missing or under-connected. The map reflects observed traffic, not declared topology.
  5. Relay not reachable / UI empty. If Relay is down or mis-deployed, the UI and hubble observe (cluster-wide mode) show nothing, even though per-node servers are fine. Diagnostic: cilium status; check the Relay Deployment. Per-node hubble observe against a single agent’s socket still works as a fallback.
  6. Metric cardinality explosion. Enabling http metrics with per-path labels on a high-cardinality API can blow up Prometheus series count. Fix: scope which metric families and labels are enabled; aggregate paths.

Alternatives and When to Choose Them

  • Service-mesh observability (Istio + Envoy sidecars, Linkerd) — a sidecar mesh produces rich per-request L7 telemetry and distributed traces, but at the cost of a proxy per Pod (CPU, memory, latency, the Sidecar Sprawl Anti-Pattern). Hubble gives you flow logs, service maps, and golden signals without sidecars, by reading eBPF events. Choose mesh observability when you need full distributed tracing and per-request mesh features (retries, mTLS policy) anyway; choose Hubble when you want network and golden-signal visibility cheaply and your data plane is already Cilium. Cilium Service Mesh and Ambient Mesh both move toward the sidecarless model Hubble exemplifies.
  • OpenTelemetry on Kubernetes — application-level traces, metrics, and logs via SDK instrumentation or auto-instrumentation. OTel sees inside the application (business spans, custom attributes) — which Hubble, observing at the network layer, cannot. Hubble sees between applications (who-talks-to-whom, policy verdicts) without touching app code. They are complementary: OTel for application internals, Hubble for the network and security layer. A mature observability stack runs both.
  • Prometheus on Kubernetes + kube-state-metrics — cluster and workload state metrics. Hubble’s metrics exporter feeds Prometheus with network golden signals; they compose rather than compete.
  • tcpdump / packet capture — ad-hoc, deep, but no Kubernetes context (just IPs), high overhead, and no policy-verdict view. Hubble is the Kubernetes-native, identity-aware, always-on replacement for the “let me tcpdump it” reflex.
  • Cloud VPC flow logs — coarse, IP-based, no Kubernetes identity, delayed. Hubble is finer, identity-aware, and real-time.

Production Notes

  • Hubble is the observability half of Cilium’s value proposition. Teams adopt Cilium for the eBPF data plane and policy; Hubble is what makes that data plane legible. On managed platforms the integration is surfaced for operators: GKE Dataplane V2 can export Hubble-derived flow logs to Cloud Logging; AKS and others expose similar Hubble-backed observability.
  • The policy-verdict view changes how network-policy debugging feels. Before Hubble, “why is this connection blocked” meant reading iptables chains. With Hubble it is hubble observe --verdict DROPPED. Teams that adopt NetworkPolicy seriously cite this as the feature that made strict default-deny policies operationally tolerable.
  • Treat flow logs as best-effort. The ring-buffer-drop behavior is by design — Hubble prioritizes low overhead over completeness. For compliance-grade records, use the exporter and accept the cost, or rely on Kubernetes audit logging for the API layer; Hubble is for operational visibility.
  • Golden signals without instrumentation is the headline. The FOSDEM 2023 “Golden Signals with Cilium and Grafana” talk (archive.fosdem.org) is the canonical demonstration: RED metrics per service pair, dashboards, and transient-issue detection — all from eBPF flow observation, no SDK in any service.
  • Pairs with Tetragon for security observability. Hubble covers network flows; Cilium’s sibling project Tetragon covers process- and syscall-level runtime security events via eBPF. Together they give network + runtime visibility from one eBPF foundation. See the Cilium production notes.

See Also

Graph View