vault

Node Problem Detector

12 min read

Node Problem Detector

node-problem-detector (NPD) is a Kubernetes add-on — a DaemonSet running one Pod on every node — whose single job is to detect node-level problems the kubelet does not surface on its own, and report them to the API server as Node conditions and/or Events (kubernetes/node-problem-detector). The kubelet already reports a handful of node conditions — Ready, MemoryPressure, DiskPressure, PIDPressure — but it is blind to a large class of infrastructure faults: a kernel deadlock, a filesystem that has gone read-only, an unresponsive container runtime, a failing NTP daemon, bad hardware. Those faults make a node unhealthy in ways the scheduler cannot see, so Pods keep landing on a broken node. NPD closes that visibility gap. Crucially, NPD is a detector, not a remediator — it observes and reports; the act of actually fixing or replacing a bad node is a separate concern handled by downstream consumers of NPD’s signals (monitor-node-health docs). It is enabled by default on GKE and AKS.

Mental Model

NPD sits between the raw node (kernel logs, journald, system metrics) and the Kubernetes API, translating low-level fault evidence into the two API objects the rest of the cluster already knows how to consume.

flowchart LR
    subgraph NODE["Each node — NPD DaemonSet Pod"]
        K[kernel log / dmesg]
        J[journald / systemd]
        M[system metrics]
        subgraph NPD["node-problem-detector"]
            SLM[system-log-monitor]
            SSM[system-stats-monitor]
            CPM[custom-plugin-monitor]
            HC[health-checker]
        end
    end
    K --> SLM
    J --> SLM
    M --> SSM
    K -.-> NPD
    NPD -->|"PERMANENT problem"| COND["Node condition<br/>e.g. KernelDeadlock=True"]
    NPD -->|"TEMPORARY problem"| EVT["Event<br/>informational, time-limited"]
    COND --> API[(kube-apiserver)]
    EVT --> API
    API -.->|"signals consumed by"| REM["REMEDIATION (separate):<br/>Draino · Karpenter/CA unhealthy-node<br/>handling · cloud auto-repair · MachineHealthCheck"]

What this diagram shows. The problem-daemon plugins (left) watch different evidence sources on the node. NPD classifies what it finds into one of two report types — permanent problems become Node conditions, temporary problems become Events — and writes them to the kube-apiserver. The dashed arrow on the right is the load-bearing boundary: NPD stops at reporting. Whatever drains, reboots, or replaces the node is a different system that consumes NPD’s conditions and events. The insight to extract: NPD turns invisible kernel/hardware faults into first-class, schedulable-relevant API signals — and deliberately leaves the dangerous part (acting on them) to a separate, explicitly-configured remediator.

Mechanical Walk-through

The two report types — and why the distinction matters

NPD reports through two distinct API mechanisms, and the choice between them is semantic, not cosmetic (NPD README):

  • Permanent problem → a Node condition. A permanent problem is one that makes the node unsuitable for running Pods until something intervenes — a kernel deadlock, a corrupted (read-only) root filesystem. NPD surfaces these as a custom Node condition (for example, a KernelDeadlock or ReadonlyFilesystem condition set to True). A Node condition is durable — it persists on the Node object until cleared — and it is visible to the scheduler and to remediation controllers, which is precisely what lets a downstream system act on it.
  • Temporary problem → an Event. A temporary problem has limited, informational impact — a transient warning that does not by itself condemn the node. NPD emits these as Kubernetes Events attached to the Node, where they show up in kubectl describe node and kubectl get events for human diagnosis but do not create a durable condition.

The reason the split matters: a condition is a machine-actionable signal a remediator can key off; an event is a human-readable breadcrumb. NPD is careful not to condemn a node for a transient blip.

Exporters — where the reports actually go

The permanent/temporary classification is what NPD found; the exporter is where it sends it. NPD ships three exporters and a problem can be routed to more than one (Monitor Node Health):

  • Kubernetes exporter — the default. Posts to the kube-apiserver: temporary problems become Events, permanent problems become Node conditions on the Node object. This is the exporter that makes faults scheduler- and remediation-visible.
  • Prometheus exporter — exposes problems and the system-stats metrics locally as Prometheus/OpenMetrics counters on an HTTP endpoint (default 127.0.0.1:20257), so a Prometheus scrape can alert on, e.g., a rising problem_counter without ever touching the API server. This is how NPD feeds dashboards and alerting in parallel with conditions.
  • Stackdriver exporter — pushes to Google Cloud Monitoring’s API, used in GKE/GCP environments.

The exporter layer is why “NPD reports conditions and events” is an under-description: those are the Kubernetes-exporter destinations specifically. A cluster can run NPD purely as a Prometheus metrics source, purely as a condition-setter, or both at once.

The problem-daemon plugin model

NPD is not a monolith; it runs a set of pluggable problem daemons, each watching a different evidence source (pkg.go.dev/k8s.io/node-problem-detector):

  • system-log-monitor — tails system logs (kernel log / dmesg, journald, or arbitrary files) and matches configured regex rules to detect log-evidenced faults: KernelDeadlock, ReadonlyFilesystem, OOM events, task hangs. The most-used daemon.
  • system-stats-monitor — collects node health metrics (disk, CPU, memory stats) and exposes them as performance data, useful for trend detection rather than discrete fault matching.
  • custom-plugin-monitor — runs arbitrary user-supplied check scripts on a schedule and turns their exit codes into conditions/events. This is the open-ended extension point: an NTP-drift check, a hardware-health probe, a vendor-specific diagnostic — anything you can script.
  • health-checker — actively checks the health of the kubelet and the container runtime themselves, catching the case where the very daemons that run Pods have hung.

Each daemon is enabled and configured through JSON configuration files mounted into the NPD Pod, specifying which logs/scripts to watch and the rules that map evidence to a named condition or event.

Where NPD stops — remediation is separate

NPD’s deliberate scope boundary is the single most important thing to understand about it. NPD detects and reports; it does not cordon, drain, reboot, or delete a node. The README has a name for the thing that does act: a remedy system is “a process or processes designed to attempt to remedy problems detected by the node-problem-detector … observe events and/or node conditions emitted by the node-problem-detector and take action to return the Kubernetes cluster to a healthy state” (NPD README). Real remedy systems include:

  • Draino — a Planet Labs project that cordons and then drains nodes matching configured labels and node conditions, optionally feeding the Cluster Autoscaler to terminate the drained node (Cloudflare’s “Automatic Remediation of Kubernetes Nodes” documents exactly this NPD-then-Draino pipeline). Draino is deliberately minimal and now largely superseded by the Descheduler-plus-TaintNodesByCondition combination and by purpose-built operators, but it remains the clearest illustration of the pattern.
  • medik8s Node Health Check Operator — the modern, actively-maintained answer: it checks each Node’s NodeConditions against thresholds in a NodeHealthCheck custom resource and, when a node is deemed failed, instantiates a remediation CR (Self Node Remediation, or fencing via Cluster API / OpenShift Machine API / a BMC / a watchdog) (medik8s/node-healthcheck-operator). This is condition-driven remediation done declaratively.
  • Cluster API MachineHealthCheck — declarative node-health remediation keyed off conditions, replacing the backing Machine when a node stays unhealthy past a timeout.
  • Cluster Autoscaler / Karpenter unhealthy-node handling — these node-lifecycle controllers can observe an unhealthy node and replace it.
  • Cloud-provider node auto-repair — GKE node auto-repair, EKS managed-node-group node auto-repair (which monitors node health and replaces nodes automatically), and AKS node auto-repair consume node health signals (NPD’s among them) to recreate bad nodes (EKS node health docs).

The “detect” and “remediate” split is intentional: detection is safe (it only writes API objects), remediation is destructive (it evicts workloads and destroys nodes), so the two are kept as separately-owned, separately-configured systems.

A widely-repeated claim is that “NPD itself gained automatic repair for FailedCgroupRemoval in v1.32.” That claim is a conflation and the correction is instructive. The auto-repair-on-FailedCgroupRemoval feature is a Google Distributed Cloud (GDC) capability — “Starting in version 1.32, when Node Problem Detector discovers select NodeConditions, it can automatically repair the corresponding problem on the node … As of version 1.32, the only NodeCondition that supports automatic repair is FailedCgroupRemoval” (GDC node-problem-detector docs). The “1.32” there is the GDC software version, not an NPD release, and the repair logic is a GDC-specific wrapper around NPD’s conditions — not a feature of the upstream kubernetes/node-problem-detector binary, whose README still frames remediation entirely as the job of an external remedy system (NPD README). So the strict “NPD detects, something else remediates” boundary holds for upstream NPD; only the GDC distribution bolts limited repair on top. Note also that NPD adopted a Kubernetes-aligned versioning scheme in late 2024 — it jumped from v0.8.x straight to v1.34.0 (September 2024), so a tag like v1.35.x tracks the Kubernetes 1.35 cycle rather than being the 135th release (NPD releases).

Configuration / API Surface

A representative NPD DaemonSet and a system-log-monitor rule:

apiVersion: apps/v1
kind: DaemonSet                          # one NPD Pod per node — node-scoped observability
metadata: { name: node-problem-detector, namespace: kube-system }
spec:
  template:
    spec:
      hostNetwork: true
      containers:
        - name: node-problem-detector
          image: registry.k8s.io/node-problem-detector/node-problem-detector:v1.35.2   # K8s-aligned versioning (≈ k8s 1.35); v0.8.x line still patched in parallel
          command:
            - /node-problem-detector
            - --config.system-log-monitor=/config/kernel-monitor.json   # enable a problem daemon
          volumeMounts:
            - { name: log,    mountPath: /var/log,    readOnly: true }   # read host kernel log
            - { name: config, mountPath: /config,     readOnly: true }   # JSON rule files
      volumes:
        - { name: log,    hostPath: { path: /var/log } }
        - { name: config, configMap: { name: npd-config } }
      tolerations:
        - operator: Exists                # run on every node, incl. tainted ones
// kernel-monitor.json — a system-log-monitor rule set (ConfigMap "npd-config")
{
  "plugin": "kmsg",
  "source": "kernel-monitor",
  "conditions": [
    { "type": "KernelDeadlock", "reason": "KernelHasNoDeadlock",
      "message": "kernel has no deadlock" }      // the DEFAULT (healthy) condition value
  ],
  "rules": [
    { "type": "permanent",                       // permanent → sets the Node CONDITION
      "condition": "KernelDeadlock",
      "reason": "AUFSUmountHung",
      "pattern": "task .* blocked for more than \\d+ seconds\\." },
    { "type": "temporary",                       // temporary → emits an EVENT only
      "reason": "OOMKilling",
      "pattern": "Killed process \\d+ (.+) total-vm:.*" }
  ]
}

Line-by-line. The DaemonSet with broad tolerations guarantees an NPD Pod on every node, including tainted ones — node faults must be detected everywhere. hostPath mounts of /var/log give NPD read access to the host’s kernel log. The JSON conditions block declares the default (healthy) value of the KernelDeadlock condition NPD will manage. The rules are the heart: each rule’s type decides the report mechanism — "permanent" flips the named Node condition, "temporary" emits a one-off Event — and pattern is the regex matched against the log source. A match on the permanent deadlock rule sets KernelDeadlock=True on the Node object, where a remediator can see it.

Failure Modes

NPD is itself a workload on the node, so it has its own failure modes:

  1. NPD silently down on a node. If the NPD Pod crashes or is evicted on a node, that node’s faults go undetected — the visibility gap reopens silently. Monitor NPD’s own liveness, and alert on missing NPD Pods.
  2. Regex rules that don’t match the actual kernel. NPD’s system-log-monitor rules are regexes against specific kernel/distro log formats. A different kernel version or distro can change the message text so the rule never fires — NPD reports a healthy node that is, in fact, deadlocked. Rules must be validated against the actual node OS.
  3. Detection without remediation. NPD faithfully sets KernelDeadlock=True, but if no remediator is wired up, the condition just sits there — workloads keep running on (or scheduling near) a broken node. NPD without a downstream consumer is a tree falling in an empty forest.
  4. False positives condemning healthy nodes. An over-broad regex or a flaky custom-plugin check can set a condition on a healthy node; if a remediator is aggressive, it will drain and destroy good capacity. Tune rules conservatively and prefer temporary/Event for anything uncertain.

Alternatives and When to Choose Them

  • The kubelet’s built-in conditions — Ready, MemoryPressure, DiskPressure, PIDPressure (Node Pressure Conditions). These are free (no add-on) but cover only resource pressure and basic reachability — they are blind to kernel/hardware/runtime faults. NPD is complementary: it adds the conditions the kubelet cannot produce.
  • Cloud-provider node health checks — GKE/EKS/AKS each run their own node-health monitoring. On a managed cluster these often include NPD (GKE and AKS enable it by default), so “do I need NPD?” may already be answered yes-by-default.
  • General node monitoring (Prometheus node_exporter) — exposes node metrics for dashboards and alerting, but produces metrics, not Kubernetes Node conditions; it does not feed scheduler/remediation decisions the way NPD does. Run both: node_exporter for trends and dashboards, NPD for actionable conditions.

Choose NPD whenever you need kernel/hardware/runtime faults to become first-class, scheduler- and remediation-visible signals — which is essentially every production cluster. Pair it with a remediator (Node Lifecycle Management) to close the loop.

Production Notes

  • Cloudflare’s “Automatic Remediation of Kubernetes Nodes” (blog.cloudflare.com) is the canonical end-to-end write-up: NPD detects the fault → sets a Node condition → Draino cordons and drains the node → the autoscaler terminates it. It is the textbook NPD-plus-remediator pipeline and the clearest illustration of the detect/remediate split.
  • GKE and AKS enable NPD by default; on a self-managed cluster it is an explicit add-on to install. On managed clusters, NPD’s signals are typically already plumbed into the provider’s node auto-repair.
  • The most common operational mistake with NPD is treating it as a fix rather than a sensor — deploying it, seeing conditions appear, and assuming bad nodes will be handled. They will not be, unless a remediator is also deployed and configured. NPD is half of a system; Node Lifecycle Management is the other half.
  • Rule maintenance is real ongoing work: kernel and distro upgrades change log message formats, so NPD’s regex rules drift out of date and must be re-validated as part of node-image upgrades.

See Also

Graph View

Backlinks