vault

Prometheus on Kubernetes

13 min read

Prometheus on Kubernetes

Prometheus is the de facto metrics and alerting system of the cloud-native world — a CNCF graduated project (accepted to the CNCF at incubating level on 9 May 2016 and promoted to graduated on 9 August 2018, the second project to graduate after Kubernetes, which graduated in March 2018) (CNCF — Prometheus) — and running it on Kubernetes has a canonical shape (prometheus.io). Prometheus works by a pull model: a central server periodically scrapes HTTP /metrics endpoints exposed by targets, parses the plaintext/OpenMetrics samples, and stores them as time series in its local TSDB. On Kubernetes, the recommended way to run it is the Prometheus Operator — a set of Custom Resource Definitions that make Prometheus, Alertmanager, and the scrape configuration themselves declarative Kubernetes objects, so a Prometheus resource and a few ServiceMonitor resources replace hand-written prometheus.yml. The whole stack — operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and a curated set of default dashboards and alerts — is packaged as the kube-prometheus-stack Helm chart, which is what most teams actually deploy. This note covers the pull model, the operator’s CRDs, how service discovery works via the Kubernetes API, and the scaling story (federation, Thanos, Mimir, remote-write) once a single Prometheus is no longer enough.

Mental Model

flowchart TB
    subgraph OP["Prometheus Operator (a controller)"]
        WATCH["watches Prometheus / ServiceMonitor /<br/>PodMonitor / PrometheusRule / Alertmanager CRs"]
        GEN["generates prometheus.yml +<br/>rule files into a Secret"]
    end
    subgraph PROM["Prometheus StatefulSet (managed by the operator)"]
        SD["K8s service discovery"]
        SCRAPE["scrape loop — pull /metrics"]
        TSDB[("local TSDB")]
        RULES["rule evaluation"]
    end
    SM["ServiceMonitor CRs<br/>'scrape pods behind this Service'"]
    TARGETS["app Pods + node-exporter + KSM<br/>exposing /metrics"]
    AM["Alertmanager StatefulSet"]
    GRAF["Grafana"]
    WATCH --> GEN --> PROM
    SM -. selected by .-> WATCH
    SD --> SCRAPE
    SCRAPE -- "HTTP GET /metrics" --> TARGETS
    SCRAPE --> TSDB
    RULES --> TSDB
    RULES -- "firing alerts" --> AM
    GRAF -- "PromQL queries" --> TSDB

What this diagram shows. Two control loops stacked. The Prometheus Operator is itself a controller: it watches its CRDs and generates the Prometheus configuration into a Secret, then makes the Prometheus StatefulSet reload it. Prometheus then runs its own loop: discover targets via the Kubernetes API, scrape their /metrics, store samples, evaluate rules, and fire alerts to Alertmanager. The insight to extract: the operator’s job is to turn declarative Kubernetes objects into imperative Prometheus config. You never edit prometheus.yml; you create a ServiceMonitor and the operator translates it. This is the same “extend the platform with CRDs and a controller” story as every other operator — Prometheus monitoring becomes just more Kubernetes resources.

Mechanical Walk-through

The pull model

A Prometheus target is anything exposing an HTTP endpoint — conventionally /metrics — that returns metrics in Prometheus text format (metric_name{label="v"} 42). Prometheus scrapes each target on an interval (commonly 15–30 s), appends the samples to its TSDB tagged with the scrape timestamp, and that is the entire ingestion path. Pull (rather than push) has deliberate consequences: Prometheus knows which targets should exist, so a target that stops responding is itself a signal (the synthetic up metric goes to 0); there is no central queue to overflow; and any target can be scraped ad hoc for debugging. The exceptions — short-lived batch jobs that vanish before a scrape — push to a Pushgateway, which Prometheus then scrapes.

The Prometheus Operator and its CRDs

Configuring raw Prometheus means hand-writing scrape_configs with kubernetes_sd_configs and intricate relabel_configs. The Prometheus Operator (prometheus-operator.dev) replaces that with CRDs. The principal kinds:

  • Prometheus — declares a Prometheus instance: replica count, retention, resources, which ServiceMonitor/PodMonitor/PrometheusRule objects it should pick up (via label selectors). The operator renders this into a managed StatefulSet.
  • Alertmanager — declares an Alertmanager cluster (routing and notification of alerts), likewise rendered into a StatefulSet.
  • ServiceMonitor — the workhorse. Declares “scrape the Pods behind Services matching this label selector.” You attach a ServiceMonitor to your application’s Service and the operator generates the corresponding scrape config — no relabeling by hand.
  • PodMonitor — like ServiceMonitor but selects Pods directly, for workloads with no fronting Service.
  • PrometheusRule — declares alerting rules (alert: — fire when a PromQL expression holds) and recording rules (record: — precompute an expensive query into a new series).
  • Probe — declares blackbox-probe targets (ping a URL/host via the blackbox exporter) — synthetic monitoring of endpoints rather than scraping their own metrics.
  • AlertmanagerConfig — namespaced alert-routing configuration, so teams can own their own routing without editing the cluster-wide Alertmanager config.
  • ScrapeConfig — an escape hatch for scrape targets not expressible as a ServiceMonitor/PodMonitor (e.g. external static targets, other SD mechanisms).
  • ThanosRuler — rule evaluation against a Thanos/Prometheus query backend at scale.
  • PrometheusAgent — Prometheus in agent mode: it scrapes and remote-writes but does not store or query locally — for forwarding metrics to a central long-term backend.

Service discovery via the Kubernetes API

Prometheus discovers what to scrape by querying the Kubernetes API itself (kubernetes_sd_configs), which can enumerate Nodes, Services, Pods, Endpoints/EndpointSlices, and Ingresses. The operator builds these SD blocks from your ServiceMonitor/PodMonitor selectors, so as Pods come and go the scrape target set updates automatically — there is no static target list to maintain. Relabeling is the glue: relabel_configs filter and rewrite target labels before scraping (drop targets, build the __address__, attach namespace/pod labels); metric_relabel_configs filter or rewrite samples after scraping but before storage (drop high-cardinality noise). The operator generates most relabeling for you; you only touch it for fine-tuning.

kube-prometheus and kube-prometheus-stack

Assembling the operator, Prometheus, Alertmanager, Grafana, exporters, RBAC, and a sensible default set of dashboards and alert rules by hand is substantial work. Two projects pre-assemble it:

  • kube-prometheus — the upstream jsonnet-based project from the Prometheus Operator org: opinionated manifests bundling the operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and a curated set of PrometheusRule alerts and Grafana dashboards covering the cluster itself.
  • kube-prometheus-stack — the Helm chart (from prometheus-community/helm-charts) that packages the same components, configurable via a values.yaml. It is the most common way teams install monitoring on Kubernetes: one helm install yields the operator, a Prometheus, an Alertmanager, Grafana with auto-loaded dashboards, node-exporter as a DaemonSet, kube-state-metrics, and the default alert rules.

Configuration / API Surface

The two CRDs a developer touches most — a ServiceMonitor and a PrometheusRule:

# Tell Prometheus to scrape an application's Pods (via its Service).
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  labels:
    release: kube-prometheus-stack   # the Prometheus CR selects ServiceMonitors by label
spec:
  selector:
    matchLabels:
      app: payments-api              # match the Service to scrape
  endpoints:
    - port: metrics                  # the Service port NAME exposing /metrics
      interval: 30s
      path: /metrics
---
# Declare an alerting rule as a Kubernetes object.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-api-alerts
  labels:
    release: kube-prometheus-stack   # likewise selected by label
spec:
  groups:
    - name: payments-api
      rules:
        - alert: PaymentsApiHighErrorRate
          expr: |                    # PromQL — fires when the expression yields data
            sum(rate(http_requests_total{job="payments-api",code=~"5.."}[5m]))
              / sum(rate(http_requests_total{job="payments-api"}[5m])) > 0.05
          for: 10m                   # must hold 10m before firing (debounce)
          labels: { severity: critical }
          annotations:
            summary: "payments-api 5xx rate above 5%"

Line-by-line, the load-bearing points:

  • monitoring.coreos.com/v1 — the Prometheus Operator’s API group (the coreos.com domain is historical; the operator originated at CoreOS).
  • The release/label on each CR — the Prometheus CR has a serviceMonitorSelector and ruleSelector; only CRs whose labels match are picked up. This label is how a ServiceMonitor gets attached to a Prometheus instance — forgetting it is the most common “my metrics never appear” mistake.
  • spec.selector.matchLabels on the ServiceMonitor — selects which Service to scrape; the operator then resolves that Service’s Endpoints to the actual Pod targets.
  • endpoints[].port — the name (not number) of the port exposing /metrics; it must match a named port on the Service.
  • PrometheusRule.spec.groups[].rules[] — alert: rules fire to Alertmanager; record: rules precompute series. for: debounces — the condition must hold continuously before the alert fires.

The scaling story

A single Prometheus is vertically bounded: its TSDB lives on one node’s disk, retention is typically days-to-weeks, and it is a single point of failure. The escalation path:

  • HA pairs — run two identical Prometheus replicas scraping the same targets; Alertmanager deduplicates their alerts. Survives one replica failing, but the two TSDBs still diverge slightly and neither holds long-term history.
  • Federation — a higher-level Prometheus scrapes aggregated series from many lower-level Prometheis. Workable for a coarse global view; does not scale to full-fidelity long-term storage.
  • Thanos — a sidecar uploads each Prometheus’s TSDB blocks to object storage; a Thanos Query layer fans out across sidecars and the object store, deduplicating HA-replica data, to give a single global, long-retention view (thanos.io). The operator’s ThanosRuler integrates rule evaluation. Thanos can also ingest push-style via Thanos Receive, which terminates Prometheus remote-write. Thanos is a CNCF Incubating project (sandbox→incubating on 19 August 2020) (CNCF — Thanos).
  • Grafana Mimir / Cortex — horizontally scalable, multi-tenant long-term backends that ingest via remote-write: Prometheus (or a PrometheusAgent) streams every sample to the central system, which owns storage and query. Cortex is the older of the pair and is a CNCF Incubating project (welcomed at incubating level in August 2020) (CNCF — Cortex). Grafana Mimir was launched by Grafana Labs in March 2022, combining the work Grafana engineers had contributed to Cortex with features from the proprietary Grafana Enterprise Metrics product; it is widely described as a Cortex fork and is licensed AGPLv3 (not Apache-2.0, so it is not a CNCF project, unlike Cortex) (Grafana — Announcing Mimir, grafana.com/oss/mimir). Mimir is the more actively marketed of the two today, but both remain in use.
  • Remote-write — the common primitive under Mimir/Cortex (and Thanos Receive): instead of pulling, Prometheus pushes its samples to a remote endpoint, decoupling collection from long-term storage (Prometheus — remote write). The architectural fork in the road is therefore pull-then-ship-blocks (Thanos sidecar → object storage) versus push-every-sample (remote-write → Mimir/Cortex/Thanos-Receive); the former keeps each Prometheus authoritative for recent data and is cheaper to bolt onto an existing fleet, while the latter centralizes ingestion and is the model the multi-tenant backends are built around.

Failure Modes

  • ServiceMonitor ignored — metrics never appear. The ServiceMonitor’s labels do not match the Prometheus CR’s serviceMonitorSelector, or it lives in a namespace the selector excludes, or the port name does not match the Service. Symptom: the target is absent from Prometheus’s /targets. Fix: align the label and namespace selectors.
  • Cardinality explosion. A metric labelled with an unbounded value (a user ID, a full URL path, a UUID) creates a near-infinite number of time series; Prometheus memory and disk balloon and it can OOM. Symptom: Prometheus restart loop, slow queries. Fix: drop the offending label with metric_relabel_configs; fix the instrumentation.
  • TSDB disk exhaustion / retention. Local TSDB is bounded by disk and retention; a single Prometheus simply cannot hold months of high-cardinality data. Symptom: oldest data silently dropped, or disk full. Fix: this is the cue to move to Thanos or Mimir.
  • Scrape interval vs rule window mismatch. A rate() over [1m] with a 60 s scrape interval has too few samples and yields gaps. Fix: keep rule ranges comfortably larger than the scrape interval (the “at least 4× the interval” rule of thumb).
  • Pull cannot reach the target. A NetworkPolicy, a missing named port, or an app not actually exposing /metrics makes the target’s up metric 0. Symptom: up == 0 for that target. Fix: that is itself an alertable signal — alert on up == 0.
  • HA replicas diverge. Two Prometheus replicas scrape at slightly different instants, so their series are not byte-identical; querying one vs the other shows small differences. Expected — Thanos Query’s deduplication is what reconciles them for a unified view.

Alternatives and When to Choose Them

  • kube-state-metrics + Metrics Server — not alternatives; inputs and neighbors. kube-state-metrics is one of Prometheus’s most important scrape targets (object-state metrics); Metrics Server is a separate lightweight resource-metrics API for the HPA (see Metrics Server for why it exists apart from Prometheus). Prometheus is the monitoring system; the other two feed or sit beside it.
  • OpenTelemetry on Kubernetes — the vendor-neutral telemetry framework, strongest on traces and logs and increasingly capable on metrics. OTel and Prometheus interoperate (the OTel Collector can scrape Prometheus endpoints and remote-write to Prometheus-compatible backends); many clusters run both, Prometheus for metrics and OTel for traces.
  • Datadog, Grafana Cloud, New Relic, hosted SaaS — managed observability that removes the operational burden of running and scaling Prometheus, at recurring cost and with data leaving the cluster. Choose when the team would rather not own a TSDB; many use a PrometheusAgent or the OTel Collector to forward into the SaaS while keeping Kubernetes-native collection.
  • Victoria Metrics — a Prometheus-compatible TSDB often chosen for higher ingestion efficiency and simpler long-term storage than Thanos/Mimir; a drop-in remote-write or scrape target. A reasonable alternative to the Thanos/Mimir scaling path.

The decision rule: on Kubernetes, deploy kube-prometheus-stack and you have a production-grade monitoring stack — operator, Prometheus, Alertmanager, Grafana, exporters, default dashboards and alerts — in one Helm release. Reach for Thanos or Mimir only when a single Prometheus’s retention or scale becomes the binding constraint; reach for a SaaS only when you would rather not operate the storage tier at all.

Production Notes

  • kube-prometheus-stack is the near-universal starting point. It is the most-installed monitoring Helm chart on Kubernetes; its bundled default PrometheusRule alerts and Grafana dashboards give a cluster a credible monitoring baseline on day one, and teams then layer their own ServiceMonitor/PrometheusRule objects on top.
  • ServiceMonitors make monitoring a developer concern. Because a ServiceMonitor is just a Kubernetes object shipped alongside the app’s Deployment and Service, application teams own their own scraping declaratively — no platform-team ticket to edit a central prometheus.yml. This is the operator’s biggest cultural win.
  • Cardinality is the operational risk that recurs. Nearly every production Prometheus incident is some flavor of cardinality blow-up — a label that should have been bounded was not. Reviewing instrumentation for unbounded label values, and keeping metric_relabel_configs ready to drop offenders, is standard hygiene.
  • Plan the long-term-storage tier before you need it. Local TSDB retention is days-to-weeks. Teams that need months of history almost always end up on Thanos or Mimir; deciding which, and wiring object storage and remote-write, is far less painful done deliberately than under disk-pressure.
  • Run Prometheus HA and let Alertmanager dedupe. A single Prometheus replica is a single point of failure for all alerting. Two replicas plus Alertmanager deduplication is the minimum production posture; Thanos Query then gives a unified, deduplicated read path.

See Also

Graph View

Backlinks