vault

kube-state-metrics

11 min read

kube-state-metrics

kube-state-metrics (KSM) is a service that listens to the Kubernetes API server and generates Prometheus metrics about the state of cluster objects — how many Pods are in each phase, whether a Deployment has reached its desired replica count, which Node conditions are set, how many times a container has restarted (github.com/kubernetes/kube-state-metrics). It exposes those metrics on an HTTP /metrics endpoint in Prometheus plaintext format, ready to be scraped. The project’s defining principle is fidelity: KSM is “about generating metrics from Kubernetes API objects without modification” — unlike kubectl, which applies human-friendly heuristics, KSM emits the raw object state verbatim. The note’s single most important point is the contrast with Metrics Server: Metrics Server reports resource usage (CPU and memory a Pod is consuming); kube-state-metrics reports object state (what the API server says about your objects). “Is this Pod using 200 mCPU” is a Metrics Server question; “is this Pod stuck Pending, and has its Deployment finished rolling out” is a kube-state-metrics question. KSM is the canonical data source for the second class — and therefore the foundation of almost every “alert me when something is wrong with my cluster’s objects” rule.

Mental Model

flowchart LR
    API["kube-apiserver"]
    subgraph KSM["kube-state-metrics Pod"]
        WATCH["informers: LIST+WATCH<br/>Pods, Deployments, Nodes,<br/>Jobs, PVCs, ..."]
        STORE["in-memory snapshot<br/>of object state"]
        GEN["metric generators<br/>(one transform per kind)"]
        EP["/metrics endpoint"]
    end
    PROM["Prometheus<br/>(scrapes /metrics)"]
    API -- "watch stream" --> WATCH --> STORE --> GEN --> EP
    PROM -- "HTTP GET /metrics" --> EP

What this diagram shows. KSM holds open informer watches on the object kinds it cares about, maintains an in-memory snapshot of their state, and runs a generator per kind that transforms that snapshot into Prometheus metrics — recomputed on demand each time Prometheus scrapes /metrics. The insight to extract: KSM is a stateless transform. It stores nothing durably, modifies nothing, and decides nothing — it is a pure function from “current API object state” to “Prometheus metrics.” It is not in any control loop; it never writes to the API server; it has no opinion. Restart it and it rebuilds its snapshot from the API and resumes. That purity is why its metrics are trustworthy: they are exactly what the API says, with no interpretation layered on top.

Mechanical Walk-through

What it watches and what it emits

KSM uses informers to watch dozens of built-in resource kinds — Pods, Deployments, ReplicaSets, StatefulSets, DaemonSets, Jobs, CronJobs, Nodes, Namespaces, PersistentVolumes, PersistentVolumeClaims, Services, Ingresses, and more — and (optionally) configured Custom Resources. For each object it emits a family of metrics. The metrics are deliberately low-level and label-rich: rather than one “is this Deployment healthy” boolean, KSM emits the raw fields and lets PromQL compose the question. Representative metric names:

  • kube_pod_status_phase — a gauge, value 1 for the Pod’s current phase and 0 for the others, with a phase label (Pending/Running/Succeeded/Failed/Unknown). sum(kube_pod_status_phase{phase="Pending"}) is the count of Pending Pods.
  • kube_pod_container_status_restarts_total — a counter of container restarts, per Pod/container. rate(...) over it is the canonical CrashLoop signal.
  • kube_deployment_status_replicas and kube_deployment_spec_replicas — observed vs desired replica counts; a gap means a rollout is incomplete or stuck. Companion metrics include kube_deployment_status_replicas_available and kube_deployment_status_replicas_unavailable.
  • kube_node_status_condition — per-Node, per-condition (Ready, MemoryPressure, DiskPressure, PIDPressure) with a status label; the basis for “alert when a Node is NotReady.”
  • kube_job_status_failed, kube_pod_container_resource_requests, kube_persistentvolumeclaim_status_phase, and many more — one family per object field worth observing.

KSM gives each metric an explicit stability stage — EXPERIMENTAL, STABLE, or DEPRECATED — and the project’s documented contract is that it adds, renames, and eventually removes metrics across releases as the underlying Kubernetes API fields evolve (KSM README, per-kind metric docs). The names above are stable and current as of KSM v2.18.0 (the latest release at the time of writing, May 2026, built against client-go for Kubernetes v1.34; the main branch tracks v1.35). For an exhaustive, version-correct list — and to see which metrics are still experimental — consult the per-kind metric docs for the exact KSM version you deploy, not the names memorized from an older release.

How it differs from Metrics Server — the load-bearing distinction

Metrics Serverkube-state-metrics
Answers”How much CPU/memory is this Pod using?""What does the API say about this object’s state?”
Data sourcekubelet /metrics/resource (cgroup usage)kube-apiserver (object specs and statuses)
Servesthe metrics.k8s.io aggregated APIa Prometheus /metrics endpoint
Consumed bykubectl top, the HPAPrometheus (then dashboards and alerts)
Example metricNodeMetrics usage.cpukube_pod_status_phase

They are complementary, not competing — most clusters run both. The mistake to avoid is reaching for Metrics Server to answer “how many Pods are Pending” (it cannot — it has no object-state metrics) or pointing Prometheus at metrics.k8s.io for monitoring (the wrong shape; KSM and node-exporter are the right Prometheus inputs).

What it does not do

KSM does not modify or persist anything. It is not a controller — it never reconciles, never writes. It does not do health-checking of the cluster’s components (that is the apiserver’s /healthz, etcd metrics, etc.); it reports the objects’ state. And it does not store time series — it emits an instantaneous snapshot on every scrape, and Prometheus is what turns successive snapshots into history.

Configuration / API Surface

KSM runs as a Deployment with a read-only RBAC ClusterRole and a Service that Prometheus scrapes. The interesting surface is its flags and a ServiceMonitor to wire it into Prometheus:

# Key container args on the kube-state-metrics Deployment.
- --resources=pods,deployments,nodes,jobs,statefulsets   # which kinds to watch
- --namespaces=team-a,team-b      # restrict scope (default: all namespaces)
- --metric-labels-allowlist=pods=[app,team]   # opt-in object labels as metric labels
- --shard=0                       # this replica's shard index (0-indexed)
- --total-shards=3                # total shards — see Scaling below
---
# A ServiceMonitor telling the Prometheus Operator to scrape KSM.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
spec:
  selector:
    matchLabels: { app.kubernetes.io/name: kube-state-metrics }
  endpoints:
    - port: http-metrics

Line-by-line, the load-bearing points:

  • --resources — narrows the watched kinds. Watching everything in a large cluster costs memory; restricting to the kinds you actually alert on keeps KSM small.
  • --metric-labels-allowlist — object labels are not turned into metric labels by default (that would explode cardinality). You opt specific labels in so PromQL can group by them (e.g. by team).
  • --shard / --total-shards — horizontal sharding (see Scaling). Each shard owns objects whose UID hashes to its index.
  • The ServiceMonitor — the Prometheus Operator CRD that declares “scrape the KSM Service”; this is the standard way KSM data reaches Prometheus.

Scaling

A single KSM Pod holds the entire object snapshot in memory; on a very large cluster that snapshot — and the scrape response — becomes large. KSM solves this with horizontal sharding (sharding design): set --total-shards=N (default 1, which disables sharding) and run N replicas, each with a distinct zero-indexed --shard. Each replica takes the MD5 sum of every object’s UID, modulo the total shard count, and keeps only the objects whose result equals its own shard index — a deterministic, stateless partition with no coordination between replicas.

There is one crucial caveat the naive mental model gets wrong: sharding splits the in-memory snapshot and the /metrics payload, but it does not split the watch traffic or the unmarshal cost. Every replica still opens watches on all objects of the configured kinds and unmarshals every object it receives, then discards the ones that do not hash to its shard. So in the optimal case each shard’s memory for the retained snapshot drops to roughly 1/N, but network and CPU-for-unmarshaling stay close to full per replica (the Kubernetes API has no sharded list/watch to push the filter server-side). Sharding is therefore primarily a memory-and-scrape-size lever, not a watch-load lever.

A StatefulSet-based automatic sharding mode (--pod / --pod-namespace) lets the replicas self-assign their shard index from their StatefulSet ordinal so the whole fleet is one resource instead of N Deployments — this mode is explicitly experimental and may change or be removed without notice. A DaemonSet mode (--node=$(NODE_NAME)) shards Pod metrics by node. Baseline resource needs are modest — the project cites ~250 MiB memory and ~0.1 cores per instance for a small cluster — but memory grows with object count, which is what sharding addresses.

Failure Modes

  • High cardinality from labels. Turning every object label into a metric label (or allowlisting a high-cardinality label like a per-Pod hash) multiplies the time series Prometheus must store and can overwhelm it. Symptom: Prometheus memory and disk balloon after a KSM config change. Fix: keep --metric-labels-allowlist tight.
  • Memory growth on large clusters. A single KSM replica’s snapshot scales with total object count; on a cluster with hundreds of thousands of objects it can OOM. Symptom: KSM Pod OOMKilled. Fix: shard with --total-shards, or narrow --resources/--namespaces.
  • RBAC too narrow. KSM needs get/list/watch on every kind it is told to watch; a missing rule means that kind’s metrics silently never appear. Symptom: an expected kube_* metric is absent. Fix: align the ClusterRole with --resources.
  • Confused with Metrics Server. Querying KSM for CPU usage or Metrics Server for “Pending Pod count” simply returns nothing useful — they answer different questions. Symptom: an alert that never fires or a dashboard panel that is always empty. Fix: use the right component for the question.
  • Stale metrics during a watch disruption. If KSM’s watch connection to the apiserver breaks, its snapshot can briefly lag until informers resync. Self-healing, but a reason to alert on KSM’s own up/scrape-health.

Alternatives and When to Choose Them

  • Metrics Server — the resource-usage counterpart, not a substitute. Run both: Metrics Server for kubectl top and CPU/memory HPA, KSM for object-state monitoring and alerting.
  • Scraping the apiserver or kubelet directly — Prometheus can scrape the apiserver’s own /metrics (which reports apiserver internals — request latencies, etcd object counts) but that is component health, not object state. KSM is purpose-built to expose object state cleanly and stably; reimplementing it with raw apiserver scraping and relabeling is wasted effort.
  • Writing PromQL against the apiserver’s apiserver_storage_objects — gives crude object counts but no per-object state (no phase, no condition, no replica gap). KSM is the right tool the moment you need per-object detail.
  • A custom exporter — only justified for object state KSM does not cover. For built-in kinds and most Custom Resources, KSM (with its CR-config feature) is the standard answer; a bespoke exporter is rarely worth maintaining.

The decision rule: if you run Prometheus and want to alert on or graph the state of Kubernetes objects — Pending Pods, incomplete rollouts, NotReady Nodes, crash loops — you run kube-state-metrics. It is, alongside node-exporter, one of the two near-universal Prometheus inputs in a Kubernetes cluster.

Production Notes

  • It is bundled into kube-prometheus-stack. The standard Prometheus Helm chart, kube-prometheus-stack, installs kube-state-metrics as a dependency alongside node-exporter — so most clusters get KSM the moment they deploy the monitoring stack, and the default dashboards and alert rules that ship with that chart are built on KSM metrics.
  • Most “cluster is unhealthy” alerts trace back to KSM. “Deployment has been not fully available for 15 minutes,” “Pod has been Pending too long,” “Node NotReady,” “container restarting repeatedly” — the canonical Kubernetes alerting rules are PromQL over kube_deployment_status_replicas_available, kube_pod_status_phase, kube_node_status_condition, and kube_pod_container_status_restarts_total. KSM is the data source for the everyday on-call alert set.
  • Shard before it hurts. On clusters past tens of thousands of objects, a single KSM replica becomes a memory and scrape-latency liability. Plan sharding (--total-shards with the StatefulSet auto-sharding mode) as part of scaling the cluster, not as an incident response.
  • Treat metric-name changes as a breaking change. Because KSM emits raw API state, a Kubernetes API field deprecation can ripple into a KSM metric rename. Pin the KSM version, read its changelog on upgrade, and check that alert rules and dashboards still reference live metric names.
  • Keep label allowlists deliberate. The temptation to allowlist every object label for “flexibility” is the most common KSM-induced Prometheus cardinality incident. Allowlist only the labels dashboards and alerts actually group by.

See Also

Graph View

Backlinks