vault

Cardinality and the Cost of Observability

14 min read

Cardinality and the Cost of Observability

Cardinality is the number of distinct time series a metric produces, and it is the silent killer of every metrics backend. Each unique combination of a metric name plus its label key-value pairs is a separate time series that must be stored, indexed, and held partly in memory — so a single innocent-looking label like user_id or request_id does not add one dimension, it multiplies the series count by the number of distinct users or requests, turning a handful of series into millions and bankrupting the backend. Prometheus states the rule flatly: “every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored,” and warns against labels holding “user IDs, email addresses, or other unbounded sets of values” (Prometheus naming). This note explains why cardinality is the cost driver of observability, lays out the metrics-vs-logs-vs-traces cost model that follows from it, states the load-bearing rule — high-cardinality data belongs in traces and logs, not in metric labels — and shows how exemplars and sampling bridge the gap so you can still find the one high-cardinality request you care about without paying for all of them.

Mental Model — cardinality multiplies, it doesn’t add

The trap is that adding a label feels additive but is multiplicative. A metric’s cardinality is the product of the number of distinct values across all its labels. Grafana’s worked example: a metric server_responses with a status_code label of 5 values and an environment label of 2 values has a cardinality of 5 × 2 = 10 series (Grafana 2022). Ten is fine. But now add user_id: because “Prometheus creates one series per label combination,” a user_id label produces “a lot of series for a single metric if you have a lot of users” (Grafana 2022). Ten series becomes 10 × 1,000,000 users = ten million series — from one metric, from one added label.

flowchart TD
    M["Metric: http_requests_total"]
    M --> L1["label: method<br/>(GET, POST, PUT, DELETE) = 4"]
    M --> L2["label: status<br/>(2xx,3xx,4xx,5xx) = 4"]
    M --> L3["label: user_id<br/>(unbounded ≈ 10^6)"]
    L1 --> S1["4 × 4 = 16 series<br/>✅ cheap, aggregatable"]
    L3 --> S2["4 × 4 × 10^6 = 16,000,000 series<br/>💥 cardinality bomb"]
    style S1 fill:#d5f5d5
    style S2 fill:#f5d5d5

Why one label ends the world. What it shows: the same metric costs 16 series with bounded labels, but adding a single unbounded user_id label multiplies that to sixteen million. The insight to take: cardinality is a product, so the cost of a label is set by its cardinality, not by the fact that it’s “just one more label” — and one unbounded dimension dominates all the bounded ones combined.

Why each series is expensive

A time series is not free to keep alive. Prometheus’s own guidance spells out the cost basis: “Each labelset is an additional time series that has RAM, CPU, disk, and network costs” (Prometheus instrumentation). The RAM cost is the sharp one — a time-series database keeps an in-memory index of active series (the inverted index mapping each label value to the series containing it, plus the head block of recent samples), so active-series count, not sample count, is usually what runs a Prometheus out of memory. When too many high-cardinality series arrive, you “begin to use too many resources, which can then lead to memory errors and system crashes” (Grafana 2022). And because hosted backends bill on active-series count, a cardinality spike also “cause[s] an increase in your expenses” (Grafana 2022) — the failure mode is sometimes a bill, not a crash. (The lower-level, kernel-side statement of “observation is never free” — the probe effect, the cost of a tracepoint or a kprobe — is in Observability Overhead and Safety; this note is the metrics-backend echo of the same principle.)

Prometheus turns this into concrete numeric guidance. As a rule of thumb, “keep the cardinality of your metrics below 10, and for metrics that exceed that, aim to limit them to a handful across your whole system”; a metric past ~100 cardinality should push you to “reducing the number of dimensions or moving the analysis away from monitoring and to a general-purpose processing system” (Prometheus instrumentation). Their scaling illustration is memorable: 10,000 nodes reporting filesystem metrics is ~100,000 series (manageable), but adding a per-user disk-quota dimension pushes it into “double digit number of millions” (unmanageable) (Prometheus instrumentation). The phrase to internalize from the naming guide: never use labels for “unbounded sets of values” (Prometheus naming) — bounded is the whole test. method (four verbs) is bounded; http.route (the path template /users/{id}, bounded by route count) is bounded; user_id (bounded by user count, i.e. unbounded from the metric’s view) is not.

The hidden multiplier: histogram buckets are series too

A subtlety that catches even careful teams: a histogram already multiplies cardinality by its bucket count. A classic Prometheus histogram is not one series — it is one series per bucket boundary (each _bucket{le="..."}), plus a _sum and a _count series. A histogram with, say, 12 buckets is therefore ~14 series for a single label combination. Now compose that with labels: a latency histogram labeled by service (20 values) and route (50 values) is 14 × 20 × 50 = 14,000 series before any high-cardinality label is even considered. This is why latency SLIs — which want percentiles, hence histograms — are where cardinality budgets are spent fastest, and why adding a high-cardinality label to a histogram is far more explosive than adding it to a plain counter. The mitigations are to keep bucket counts modest and label sets bounded, and (on newer Prometheus) to prefer native/exponential histograms, which represent the distribution far more compactly than a series-per-bucket classic histogram. The takeaway for capacity planning: when you estimate a metric’s cost, multiply the label-combination count by the per-observation series count — 1 for a counter/gauge, buckets+2 for a classic histogram.

Uncertain

Verify: the exact series-count formula for Prometheus native (exponential) histograms and which Prometheus/OpenMetrics version made them the recommended default. Reason: stated from general knowledge of the Prometheus data model; the native-histogram specifics were not fetched from a primary during this task. To resolve: check the Prometheus native-histograms documentation and release notes. #uncertain

The cost model of the three pillars

Cardinality is the axis that explains why the three observability signals — metrics, logs, traces — are complementary rather than interchangeable, and why each is priced the way it is. (The signals themselves are compared in The Three Pillars of Observability; here the lens is strictly cost.)

  • Metrics — cheap, aggregatable, must be low-cardinality. A metric is a pre-aggregated counter or histogram: the value of a 2xx counter is the same object no matter how many requests it counts, so cost scales with the number of series, not the number of events. That is why metrics are the substrate for SLOs and dashboards — RED gives every service a rate, an error fraction, and a latency histogram at trivial cardinality (Wilkie 2018). The price of that cheapness is the hard cardinality ceiling above: the moment you want to slice by a high-cardinality dimension, metrics are the wrong tool.
  • Logs — medium cost, per-event, high-cardinality-tolerant. A log line is one discrete event with arbitrary fields; you can put the user_id, the request_id, the raw URL, the stack trace — any high-cardinality value — because logs are stored and indexed as events, not as a series per distinct value. The cost scales with event volume (and with how much you index), which is why logs are typically sampled or retained for shorter windows than metrics.
  • Traces — sampled, per-request, high-cardinality-tolerant but incomplete. A span carries arbitrary attributes (Dapper stores rich per-span annotations, and today’s spans carry user.id, http.route, etc.), so traces also tolerate high cardinality. But traces are almost always sampled — Dapper found one trace in a thousand sufficient for common uses (Dapper §1) — so any single request may not be recorded. Traces answer where the time went for a representative (or error-selected) subset; they are not a complete ledger. (See Distributed Tracing in Practice.)
flowchart LR
    subgraph pick["Where does a dimension go?"]
        Q{"How many distinct<br/>values?"}
        Q -->|"bounded, small<br/>(status, method, route)"| MET["METRICS<br/>label it · low-card · SLOs"]
        Q -->|"unbounded<br/>(user_id, request_id, raw URL)"| HC["LOGS or TRACES<br/>field/attribute · sampled"]
    end

The routing rule. What it shows: a decision on where a new dimension belongs, keyed only on its cardinality. The insight to take: the choice of signal is not about “importance” — a user_id is important — it’s about cardinality. Bounded dimensions become metric labels; unbounded ones become log fields or span attributes, reached by search or by an exemplar, never by a metric label.

The load-bearing rule and how to keep it

High-cardinality data belongs in traces and logs, not in metric labels. This is the single rule that keeps a metrics backend solvent. Enforcing it has two layers.

Layer 1 — discipline at instrumentation time. Choose the bounded proxy for the dimension you want. Tag customer_tier (Free / Pro / Enterprise = 3 values) not customer_id; tag http.route (the route template, bounded by the number of routes) not the resolved URL with its embedded ids and query string; tag status_class (2xx/3xx/4xx/5xx = 4) not the exact status when you don’t need it. Do not embed a label’s identity into the metric name either — Prometheus warns against putting label names in metric names, since it creates redundancy that breaks when the label is aggregated away (Prometheus naming).

Layer 2 — guardrails, because discipline fails. The cardinality bomb is almost always an accident — a developer adds a request_id label “just for debugging,” and the series count detonates within a week. Practical defenses:

  • Cardinality limits at the collector/scrape layer that drop or reject series above a per-metric ceiling, so one bad metric can’t take down the backend.
  • Recording rules / pre-aggregation that roll high-cardinality raw series into the low-cardinality series dashboards actually query, so the expensive raw series can have short retention.
  • Cardinality dashboards on your own metrics backend (series-per-metric, series-per-label) so a spike is seen — cardinality spikes are dangerous precisely because they’re invisible until the OOM (Grafana 2022).

Sampling as the cost lever for the high-cardinality signals

If metrics stay cheap by dropping cardinality, traces and logs stay affordable by dropping volume — that is, sampling. The trade-off is the same one detailed in Distributed Tracing in Practice: head sampling decides “as early as possible” and is cheap but blind to which traces will matter; tail sampling decides after the fact and can keep exactly the interesting traces — errors and the slow tail — at the price of “stateful systems that can accept and store a large amount of data,” potentially “dozens or even hundreds of compute nodes” (OTel sampling). The SRE framing: metrics give you complete, cheap, low-cardinality coverage for SLOs; sampled traces/logs give you incomplete, richer, high-cardinality detail for debugging. You spend your observability budget by choosing, per dimension, which of those two regimes it lives in.

Exemplars — keeping metrics cheap while still reaching one hot example

The obvious objection to “no high-cardinality on metrics” is: but during an incident I have a p99 latency spike and I want the trace of a slow request. You don’t need the trace_id on the metric as a label — that would be the ultimate cardinality bomb (one series per request). You need an exemplar. An exemplar attaches a sample of context — an optional trace_id and span_id — to a single metric data point, existing specifically to “link Trace signals w/ Metrics” (OTel metrics data model). It does not create new series: for a histogram, the exemplar’s value already participates in the bucket counts, so there is no double counting (OTel). Prometheus stores exemplars in “a fixed size circular buffer” in memory, where a single trace_id exemplar costs about 100 bytes (Prometheus feature flags) — bounded, not per-series.

The workflow the exemplar enables is the whole point: a latency SLO’s histogram bucket fills, you click the bucket, the exemplar hands you the trace_id of an actual slow request, and you jump into its trace waterfall. The metric stayed low-cardinality; the exemplar carried the one high-cardinality pointer you needed. This is the concrete realization of “cheap aggregate for measuring, sampled detail for debugging, one bridge between them.”

Failure Modes and Diagnosis

  • The cardinality bomb (series explosion). A newly deployed label with an unbounded domain (user_id, request_id, session_id, full URL, error message string, unbounded pod name). Symptom: active-series count climbs, Prometheus RAM climbs, then OOM — or the hosted bill jumps. Diagnose by ranking metrics by series count (topk over count by (__name__) of series) and finding the metric whose count exploded; find the offending label by grouping. Fix: drop the label, relabel it away at scrape time, or route that dimension to logs/traces.
  • Slow-burn cardinality. No single spike — dozens of metrics each carry a slightly-too-high label (a pod label on a fleet that churns pods daily makes each day’s pods new series). The pod/instance labels are the classic churn source: dead pods leave stale series that still consume index memory until they age out. Bound retention and be deliberate about per-pod labels.
  • “It’s fine in staging.” Cardinality scales with production entity counts (users, tenants, routes, pods). A tenant_id label is invisible in a 3-tenant test and lethal at 50,000 tenants. Estimate the production domain size of every label before shipping it.
  • Aggregating away the label you paid for. If every dashboard sums over user_id, you paid ten million series’ worth of cost to render a graph that ignores the dimension entirely — a pure waste. If you always aggregate a dimension away, it should never have been a label.
  • Confusing “important” with “belongs on a metric.” A user_id is operationally important; that is an argument for putting it on a span attribute or log field (reachable via search and exemplars), not on a metric label. Importance is not the routing criterion — cardinality is.

A concrete diagnosis walk-through

When a Prometheus starts OOMing or a hosted bill jumps, the drill is mechanical. First, find which metric exploded by ranking metrics by their live series count — in PromQL, topk(10, count by (__name__)({__name__=~".+"})) returns the ten metric names with the most series; the offender is usually an order of magnitude above its neighbours. Second, find which label is responsible by fixing the metric name and grouping by candidate labels — count(count by (suspect_label) (the_metric)) gives the distinct-value count of suspect_label; the label whose count is in the thousands-to-millions is the bomb. Third, confirm the shape: a sudden step (a bad deploy added a label) versus a steady ramp (slow churn from pod/instance labels on a rolling fleet) points at different fixes. The remediation ladder, cheapest first: drop the label with a metric_relabel_configs labeldrop/drop at scrape time (no code change, immediate); replace the raw metric with a recording rule that pre-aggregates to the low-cardinality series dashboards use, and shorten retention on the raw series; or, if the dimension is genuinely needed at request granularity, move it out of metrics entirely into logs or span attributes reached by exemplars. The meta-lesson is that cardinality is operable only if you monitor it — a series-count-per-metric panel on your own monitoring backend turns an invisible OOM into a visible, alertable trend before it takes the backend down.

See Also

Graph View

Backlinks