vault

Node Lifecycle Management

19 min read

Node Lifecycle Management

Node lifecycle management is the practice of operating worker machines as cattle, not pets — treating every node as a fungible, disposable unit that is provisioned fresh, runs for a bounded lifetime, and is replaced rather than repaired. The lifecycle is a loop: a node is provisioned from an OS image, bootstraps and joins the cluster, serves Pods, and is eventually cordoned and drained and deleted — at which point its replacement is provisioned from a newer image. The single most consequential decision in this loop is the OS-image strategy: do you treat a node’s operating system as mutable (a long-lived machine you patch in place with config management) or immutable (a machine you never touch — to update it, you destroy it and boot a replacement from a new image)? The immutable model — embodied by purpose-built node operating systems like Bottlerocket, Talos Linux, and Flatcar (The New Stack — immutable operating systems) — has become the production default, because it converts every maintenance event (OS patch, kernel CVE, instance-type change) into the same simple operation: replace the node. This note covers that loop end-to-end; the static-sizing side lives in Capacity Planning on Kubernetes and the cluster-wide rhythm in Cluster Maintenance Patterns.

Mental Model

flowchart LR
    IMG["OS image / AMI<br/>(kubelet + runtime baked in)"] --> PROV["Provision<br/>cloud VM boots"]
    PROV --> BOOT["Bootstrap<br/>TLS bootstrap → CSR →<br/>kubelet registers Node object"]
    BOOT --> READY["Ready & Schedulable<br/>serves Pods"]
    READY --> NPD["node-problem-detector<br/>surfaces kernel/HW faults<br/>as Conditions + Events"]
    NPD -->|"problem detected"| DRAIN
    READY -->|"planned: patch / CVE /<br/>instance change / scale-down"| DRAIN["Cordon + Drain<br/>(PDB-aware eviction)"]
    READY -->|"spot reclaim:<br/>~2 min notice"| DRAIN
    DRAIN --> DELETE["Delete node<br/>terminate VM"]
    DELETE -->|"replacement booted from<br/>a NEWER image"| IMG

What this diagram shows. The lifecycle is a cycle, not a line: a deleted node’s replacement is provisioned from a newer image, so the loop is also the update mechanism. The left half is birth — image → VM → bootstrap → Ready. The right half is death — triggered three ways: a planned maintenance event, a detected fault (surfaced by node-problem-detector), or a spot reclamation. All three converge on the same operation: cordon, drain (respecting PDBs), delete. The insight to extract: in the immutable model there is no “update a node” operation — there is only “replace a node,” and replacement is update.

Mechanical Walk-through

Provisioning and bootstrap

A node begins as an OS image — a cloud AMI (Amazon Machine Image) or equivalent — with the kubelet, a CRI container runtime (containerd), and a CNI binary already baked in. The cloud provider boots a VM from that image. The VM then bootstraps into the cluster:

  1. The kubelet starts with a bootstrap kubeconfig holding a short-lived bootstrap token, and authenticates to the kube-apiserver as system:bootstrappers.
  2. It submits a CertificateSigningRequest with signerName: kubernetes.io/kube-apiserver-client-kubelet (TLS Bootstrapping). In a kubeadm cluster the controller-manager auto-approves CSRs from the bootstrap group.
  3. The kubelet receives its signed client certificate, switches to certificate auth, and creates its Node object in the API server.

From that point the node is registered, the kube-scheduler may place Pods on it, and the kubelet begins its dual heartbeat (full NodeStatus plus the lightweight kube-node-lease Lease — see Kubernetes Node). Everything in steps 1–3 is in the image or a startup script — which is precisely what makes the OS-image strategy the central question.

The node-lifecycle-controller and taint-based eviction — the control-plane side of failure

Provisioning is the kubelet’s story; failure handling is the control plane’s. The node-lifecycle-controller, one of the controllers inside the kube-controller-manager, watches every Node’s heartbeat and decides when a node has gone bad and what to do about its Pods (Nodes — node controller). The mechanism is precise and worth tracing end to end, because it determines exactly how long a Pod survives on a dead node.

The kubelet renews its kube-node-lease Lease on a short interval. The node-lifecycle-controller checks each node’s lease; if it has heard no heartbeat within --node-monitor-grace-period, it flips the node’s Ready condition to Unknown. That grace period defaulted to 40 s for years and was raised to 50 s in Kubernetes v1.32, because the 40 s value was tripping spurious NotReady flaps under load (kubernetes/kubernetes #127352). The controller itself re-checks on --node-monitor-period (default 5 s).

Once the node is NotReady (or Unknown), eviction is driven entirely by taints, not by a timer on the controller. The node-lifecycle-controller adds one of two taints, both with the NoExecute effect (Taints and Tolerations):

  • node.kubernetes.io/not-ready:NoExecute — the node reports itself not-ready (kubelet alive but Ready=False).
  • node.kubernetes.io/unreachable:NoExecute — the controller has lost the heartbeat entirely (Ready=Unknown).

NoExecute governs running Pods: a Pod that does not tolerate the taint is evicted immediately; a Pod that tolerates it with a tolerationSeconds value stays bound for that many seconds and is then evicted; a Pod that tolerates it with no tolerationSeconds stays forever. Here is the subtlety almost everyone gets wrong: the kube-apiserver’s DefaultTolerationSeconds admission controller automatically injects a 300-second toleration for both taints onto every Pod that lacks one. So an ordinary Pod is not evicted the instant its node goes unreachable — it carries an implicit 5-minute grace window (default-not-ready-toleration-seconds / default-unreachable-toleration-seconds, both 300, set on the kube-apiserver). After 300 s the toleration expires and the Pod is evicted. This 5-minute default is the real “how long until my Pods move after a node dies” answer, and it is tunable per-Pod by setting an explicit shorter tolerationSeconds.

The deprecated --pod-eviction-timeout

Older material talks about a --pod-eviction-timeout flag (default 5 m) on the controller-manager as the eviction knob. That mechanism was superseded by taint-based eviction, which has been the only path since it went on by default (around v1.13); the flag is effectively a no-op on modern clusters, and the eviction delay is now set via the kube-apiserver’s default-*-toleration-seconds flags or per-Pod tolerationSeconds (Taints and Tolerations). Reaching for --pod-eviction-timeout to tune eviction speed on a current cluster will silently do nothing.

The eviction actor itself was decoupled from the node-lifecycle-controller. As of Kubernetes v1.29 the taint-based eviction logic moved into a standalone taint-eviction-controller behind the SeparateTaintEvictionController feature gate (Beta, default-on in 1.29), and that decoupling reached Stable in v1.34 (Kubernetes 1.29 blog; Kubernetes v1.34 blog). The node-lifecycle-controller now applies the taints; the taint-eviction-controller acts on them. Functionally the eviction behavior is unchanged — the split is for maintainability and to let the eviction controller be disabled independently.

Graceful node shutdown — the kubelet-side complement

The taint-eviction path handles a node that fails; Graceful Node Shutdown handles a node that is deliberately shutting down (an OS reboot, a cloud maintenance event, an ACPI power signal). Without it, a shutdown kills containers abruptly, skipping preStop hooks and SIGTERM handling. The feature makes the kubelet aware of the system shutdown via systemd inhibitor locks: on a shutdown signal it sets the node NotReady with reason “node is shutting down,” stops admitting new Pods, and terminates running Pods in two ordered phases governed by KubeletConfiguration (Node Shutdowns):

shutdownGracePeriod: 30s              # total window the kubelet delays shutdown for Pod termination
shutdownGracePeriodCriticalPods: 10s  # of that, the slice reserved for critical (system) Pods

With these values, regular Pods get the first 20 s, critical Pods the final 10 s. As of the current releases (k8s ~1.36, as of 2026-05) Graceful Node Shutdown is still Beta but enabled by default since v1.21 on Linux (GracefulNodeShutdown feature gate; the Windows equivalent WindowsGracefulNodeShutdown went Beta/default-on in v1.34). A finer-grained variant, Pod-Priority-Based Graceful Node Shutdown (GracefulNodeShutdownBasedOnPodPriority, Beta default-on since v1.24), lets you map priority classes to per-class grace periods via shutdownGracePeriodByPodPriority. Its sibling for un-graceful loss — a node that is gone and not coming back — is Non-Graceful Node Shutdown, which reached GA in v1.28 and relies on an operator (or human) tainting the dead node with node.kubernetes.io/out-of-service:NoExecute so the control plane can force-delete its Pods and let StatefulSet replicas reschedule (Kubernetes 1.28 Non-Graceful Node Shutdown GA).

Mutable vs immutable OS images — the defining choice

Mutable nodes. The traditional model: a node runs a general-purpose Linux (Ubuntu, Amazon Linux, RHEL) and is long-lived. To patch it, you SSH in (or run a config-management tool — Ansible, Puppet) and apt upgrade / yum update in place. The node keeps its identity across patches. The problem: nodes drift. Two nodes “patched the same way” diverge over time (a failed patch here, a manual hotfix there), and the cluster becomes a collection of subtly different snowflakes — the antithesis of cattle. Debugging “why does Pod X fail on node 7 but not node 8” becomes archaeology.

Immutable nodes. The modern model: the node OS is a read-only, purpose-built image containing only what is needed to run containers — no package manager, often no SSH, no shell. You never modify a running node. To update — a kernel CVE, a kubelet bump, a config change — you build a new image and replace the node. Purpose-built node operating systems (The New Stack, Sidero Labs guide):

  • Bottlerocket (AWS) — a minimal, container-focused Linux with a read-only root filesystem and SSH disabled by default; ships as EKS-/ECS-integrated AMIs with a variant build per supported Kubernetes minor. Updates via an A/B partition scheme.
  • Talos Linux (Sidero Labs) — radically minimal (on the order of a dozen binaries), no SSH and no shell at all — the node is managed entirely through a declarative API. Runs on any cloud and bare metal. Uses an A/B partition scheme: a new image is written to the inactive partition; the node reboots into it; if the new image fails to boot, the bootloader automatically reverts to the last working partition.
  • Flatcar Container Linux — the actively-maintained successor to CoreOS Container Linux; accepted into the CNCF Incubator in October 2024, the first operating-system distribution CNCF has adopted (CNCF announcement); auto-updating with release channels (Alpha/Beta/Stable/LTS), the LTS channel cut roughly yearly for teams that cannot tolerate frequent reboots. Like the others it uses an A/B (active/passive) partition update scheme with automatic rollback.
  • Fedora CoreOS — the upstream Red Hat / community immutable OS in the same family.
  • The cloud providers’ own optimized images — the GKE Container-Optimized OS, the EKS-optimized AMIs — sit on the same spectrum: minimal, hardened, replaced rather than patched.

The immutable model’s payoff is uniformity and a single update path: every node of a given image is byte-identical, drift is impossible, and every maintenance event reduces to “replace the node.” Its cost: you cannot hotfix a node — even a one-line change means a full image build and a rolling replacement. In practice that cost is a feature, because it forces all change through a reviewable, version-controlled pipeline.

Patching cadence and CVE response

In the mutable world, patching is a recurring chore — a monthly/weekly apt upgrade sweep, with emergency out-of-band patches when a critical CVE (a kernel privilege-escalation, a glibc flaw) drops. In the immutable world, “patching” is node replacement: a new image with the fix is built, and the fleet is rolled — old nodes drained and deleted, replacements booted from the new image. CVE response in the immutable model is therefore the same operation as a routine kubelet bump, which is exactly why it is faster and less error-prone: there is one well-rehearsed procedure, not two. The fleet-wide mechanics of rolling that replacement are covered in Cluster Maintenance Patterns.

Draining — the graceful-vs-forced trade-off

Before a node is deleted — for any reason — its Pods must be moved off it. This is cordon + drain (Safely Drain a Node):

kubectl cordon node-7                              # spec.unschedulable=true: no new Pods land
kubectl drain node-7 --ignore-daemonsets \
        --delete-emptydir-data --grace-period=120  # evict existing Pods, 120s graceful window

cordon flips spec.unschedulable=true, so the scheduler skips the node — existing Pods stay, no new ones arrive. drain then evicts each Pod through the Eviction API, which is PDB-aware: if a PDB protects a workload (minAvailable: 2), drain blocks until evicting the next Pod would not violate it. This is the heart of the graceful-vs-forced trade-off:

  • Graceful drain respects PDBs and honors each Pod’s terminationGracePeriodSeconds (the --grace-period flag can override it). It can block indefinitely if a PDB is unsatisfiable — but it never causes an availability incident.
  • Forced drain (--disable-eviction, or --force for un-controlled Pods, or a short --grace-period=0) bypasses PDBs and graceful shutdown. It always completes — but it can take a service below its safe replica count. Reserved for emergencies (a node failing now) where the alternative is worse.

The disciplined default is graceful; forced is the break-glass.

Node problem detection

A node can be Ready and yet broken — a corrupting disk, a flapping NIC, a kernel that has logged an oops. The kubelet’s own conditions ( PIDPressure) catch resource exhaustion but not these lower-level faults. node-problem-detector (kubernetes/node-problem-detector) fills the gap. It is a DaemonSet — one Pod per node (DaemonSet) — that watches the node’s kernel ring buffer (kmsg), system logs (journald), and other sources against a set of rules, and reports what it finds to the API server. Its reporting convention (Monitor Node Health):

  • Permanent problems (a kernel deadlock, a corrupt filesystem, a bad disk) → reported as a custom NodeCondition on the Node object.
  • Temporary problems (a transient daemon hiccup) → reported as a Kubernetes Event.

A surfaced NodeCondition is then actionable: an auto-remediation controller (GKE node auto-repair, an operator, a cloud node-group health check) watches for the bad condition and triggers the node’s replacement — closing the loop back to “drain and delete.” This is what makes the cattle model self-healing: faults become signals, signals trigger replacement.

Spot / preemptible interruption handling

Spot (AWS) / preemptible (GCP) / Spot (Azure) nodes are deeply discounted but reclaimable by the cloud on short notice — typically a ~2-minute warning. Handling the interruption is a node-lifecycle event: a small controller (the AWS Node Termination Handler, or Karpenter’s built-in interruption handling) watches the cloud’s interruption-notice metadata endpoint and, on a notice, immediately cordons and drains the doomed node so its Pods reschedule before the VM is yanked. Without that handler, the node simply vanishes and its Pods hard-fail. Spot nodes are appropriate for stateless and batch workloads with surge capacity; their lifecycle is just the normal loop with an externally-triggered, time-boxed drain.

Configuration / API Surface

A managed node group declaring the desired image and lifecycle policy. Note an important correction: EKS managed node groups are not a Kubernetes CRD — there is no in-cluster apiVersion: eks.amazonaws.com/v1 Nodegroup object. A managed node group is an AWS resource created through the EKS API, the AWS CLI, CloudFormation, or eksctl (EKS managed node groups). The genuinely declarative form is an eksctl ClusterConfig:

apiVersion: eksctl.io/v1alpha5      # eksctl's config schema — NOT a Kubernetes API object
kind: ClusterConfig
metadata:
  name: prod
  region: us-east-1
managedNodeGroups:
  - name: general-bottlerocket
    amiFamily: Bottlerocket           # immutable, purpose-built node OS
    instanceTypes: [m6i.2xlarge]
    minSize: 3
    maxSize: 20
    desiredCapacity: 6
    updateConfig:
      maxUnavailablePercentage: 25    # roll ≤25% of the group at once when replacing
    labels:
      workload-type: general
    taints:
      - key: workload-type
        value: general
        effect: NoSchedule
    # spot: true                      # → spot nodes; EKS enables Capacity Rebalancing automatically

Line-by-line. amiFamily: Bottlerocket selects the immutable node OS — the operator never patches these instances, only replaces them. updateConfig.maxUnavailablePercentage: 25 governs the rolling replacement: when the image is bumped (a new Bottlerocket build with a CVE fix), the managed-node-group controller cordons + drains + replaces nodes in batches of at most 25% of the group, so capacity stays above 75% throughout — this is node lifecycle and Cluster Maintenance Patterns meeting. labels + taints dedicate the pool to a workload class. Setting spot: true turns the group into reclaimable instances; EKS then automatically enables EC2 Capacity Rebalancing so the group launches a replacement on a rebalance recommendation and drains the doomed node on a best-effort basis — so on EKS managed node groups you do not run a separate interruption handler (EKS managed node groups — capacity types).

The node-problem-detector DaemonSet, abridged:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-problem-detector
  namespace: kube-system
spec:
  selector: { matchLabels: { app: node-problem-detector } }
  template:
    spec:
      containers:
        - name: node-problem-detector
          image: registry.k8s.io/node-problem-detector/node-problem-detector:v1.35.2   # K8s-aligned versioning
          securityContext:
            privileged: true          # needs host kmsg / journald access
          volumeMounts:
            - { name: log,    mountPath: /var/log,    readOnly: true }
            - { name: kmsg,   mountPath: /dev/kmsg,   readOnly: true }
      volumes:
        - { name: log,  hostPath: { path: /var/log } }
        - { name: kmsg, hostPath: { path: /dev/kmsg } }

It mounts the host’s /var/log and /dev/kmsg read-only, watches them against its rule set, and posts NodeConditions/Events back to the API server.

Failure Modes

Bootstrap fails — node registers but is useless. The VM boots and the kubelet registers the Node, but image pulls fail (missing registry credentials, a blocking network ACL) — Pods stick in ImagePullBackOff. The node looks Ready but runs nothing. Detection needs Pod-status monitoring, not just node-status (see Common Kubernetes Failures Catalog).

CNI not ready — NetworkUnavailable wedge. The node registers before its CNI DaemonSet has programmed routes; NetworkUnavailable=True, and Pods scheduled there fail to get IPs. Usually self-resolves; if not, the CNI DaemonSet itself is broken.

Drain blocks forever on a PDB. A graceful drain stalls because a Pod Disruption Budget is unsatisfiable (minAvailable: 2 with only 2 replicas). Node replacement — and any maintenance behind it — halts. Fix: scale the workload up, relax the PDB, or escalate to a forced drain.

Spot reclaim with no interruption handler. A spot node is reclaimed; with no handler to drain it on the 2-minute notice, its Pods hard-fail and reschedule cold. Fix: deploy the Node Termination Handler or use Karpenter’s built-in handling.

Immutable-OS hotfix temptation. An operator needs an urgent one-line change and has no SSH (Talos) or a read-only root (Bottlerocket). Trying to circumvent immutability — re-enabling SSH, remounting root rw — defeats the model and reintroduces drift. The correct response is the full image-build-and-roll pipeline; if that pipeline is too slow for emergencies, that is the bug to fix.

Image drift on mutable nodes. Long-lived mutable nodes diverge — a missed patch, a manual fix — until “node 7 behaves differently” becomes an unsolvable mystery. The fix is structural: move to immutable images and replacement-based updates.

Alternatives and When to Choose Them

  • Managed node groups (EKS managed node groups, GKE node pools, AKS node pools). The cloud owns provisioning, the OS image, bootstrap, and rolling replacement. Choose for almost everything — the operational savings dwarf the loss of control.
  • Karpenter-managed nodes. Karpenter provisions right-sized nodes just-in-time and consolidates aggressively; its lifecycle is churnier by design (nodes are short-lived). Choose for heterogeneous, bursty, cost-sensitive workloads on a supported cloud.
  • Serverless nodes (Fargate, GKE Autopilot). Node lifecycle vanishes entirely — the provider manages compute and bills per Pod. Choose when you want zero node management and accept the per-Pod premium and constraints (limited DaemonSet support, slower starts).
  • Self-managed mutable nodes (kubeadm on general-purpose Linux). Maximum control, maximum operational burden, real drift risk. Choose only when an immutable image cannot meet a hard requirement (exotic kernel modules, specialized drivers).
  • Self-managed immutable nodes (Talos / Flatcar on bare metal or unmanaged VMs). The cattle model without a managed control plane underneath. Choose for on-prem and edge where managed node groups do not exist but drift-free operations are still wanted.

Production Notes

  • The industry consensus has settled firmly on immutable node operating systems: AWS pushes Bottlerocket as “EKS-managed nodes for the OS”; GKE defaults to Container-Optimized OS; Talos has gained ground for its API-only, SSH-free management (Sidero Labs). The common thread is no in-place mutation.
  • Talos’s A/B partition + automatic rollback is the gold-standard safety mechanism: a bad image cannot brick a node, because the bootloader reverts to the last-known-good partition. Mutable in-place patching has no equivalent — a bad apt upgrade can leave a node unbootable with no automatic recovery.
  • Pinterest’s K8s migration (2020) documented a kubelet certificate-rotation incident on inherited long-lived (10-year) bootstrap certs — a node-lifecycle hazard: cert handling must be designed in at provisioning time, not bolted on (see Kubernetes Node for the rotation mechanics).
  • The k8s.af catalog features lifecycle failures: CNI DaemonSets failing to come up after a node reboot, leaving Pods in ContainerCreating; a GKE node-auto-repair loop interacting badly with a failing admission webhook to destroy nodes faster than they could be replaced (see Common Kubernetes Failures Catalog).
  • The operating principle that ties it together: a node is a build artifact, not a server. If you find yourself SSH-ing into a node to “fix” it, the lifecycle has already failed — the fix belongs in the next image.

The volatile facts in this note are verified as of 2026-05: AWS issues a Spot interruption notice two minutes before reclaiming an instance, with an earlier rebalance recommendation signal arriving sooner when EC2 sees elevated interruption risk (EC2 Spot interruption notices; EC2 rebalance recommendations). Bottlerocket’s update model is confirmed dual-partition (active/inactive partition sets, swap-on-update, automatic rollback if the new image fails to boot), orchestrated in-cluster by the Bottlerocket Update Operator “Brupop” (bottlerocket-os/bottlerocket). Flatcar was accepted into the CNCF Incubator in October 2024 — the first OS distribution CNCF has taken on (CNCF: Flatcar brings Container Linux to the CNCF Incubator).

See Also

Graph View

Backlinks