vault

DaemonSet

17 min read

DaemonSet

A DaemonSet is the Kubernetes API object that ensures a Pod runs on every matching node. As nodes join the cluster, the DaemonSet controller adds a Pod to them; as nodes leave, those Pods are garbage-collected. The DaemonSet is the canonical primitive for per-node infrastructure — log shippers (Fluent Bit, Vector, Fluentd), monitoring agents (node-exporter, the Datadog Agent, the New Relic Infrastructure agent), CNI dataplane components (Cilium agent, Calico Felix, the AWS VPC CNI, kube-proxy itself on most managed K8s), CSI node plugins (one per node to manage volume mounts), and runtime-level extensions (the NVIDIA GPU device plugin, security agents like Falco). The Kubernetes documentation enumerates the canonical use cases plainly: “running a cluster storage daemon on every node, running a logs collection daemon on every node, running a node monitoring daemon on every node” (kubernetes.io — DaemonSet). The DaemonSet’s distinguishing characteristic from Deployment is that its scaling axis is the node set, not the replica count — a DaemonSet with no node selector runs as many Pods as there are eligible nodes, and that number changes automatically as the cluster autoscaler (Cluster Autoscaler, Karpenter) adds or removes nodes. The DaemonSet is also distinctive in its tolerance behavior: DaemonSet Pods receive an opinionated set of default tolerations from the controller that let them survive node-condition taints which would evict any normal Pod, so the log shipper stays running on a node experiencing memory pressure precisely when you most need its logs.

Mental Model

flowchart TB
    DS[DaemonSet 'fluent-bit'<br/>spec.template.spec.tolerations: ...<br/>spec.selector: app=fluent-bit]
    DSCTL[DaemonSet Controller<br/>watches Nodes + Pods]
    SCHED[kube-scheduler<br/>default scheduler binds]

    N1[Node node-1<br/>labels: zone=us-east-1a<br/>taints: none]
    N2[Node node-2<br/>labels: zone=us-east-1b<br/>taints: dedicated=gpu:NoSchedule]
    N3[Node node-3<br/>labels: zone=us-east-1c<br/>NotReady, MemoryPressure]
    NEW[Node node-4<br/>just joined]

    P1[Pod fluent-bit-abc<br/>nodeAffinity: name=node-1]
    P2[Pod fluent-bit-def<br/>nodeAffinity: name=node-2<br/>tolerates dedicated=gpu]
    P3[Pod fluent-bit-ghi<br/>nodeAffinity: name=node-3<br/>tolerates not-ready, memory-pressure]
    P4[Pod fluent-bit-jkl<br/>nodeAffinity: name=node-4]

    DS --> DSCTL
    DSCTL -- "for each eligible node:<br/>create Pod with<br/>nodeAffinity = name pinning" --> P1
    DSCTL --> P2
    DSCTL --> P3
    DSCTL -- "node joins → new Pod" --> P4
    P1 -- "scheduler binds" --> N1
    P2 --> N2
    P3 --> N3
    P4 --> NEW
    SCHED -. binds .-> P1
    SCHED -.-> P2
    SCHED -.-> P3
    SCHED -.-> P4

What this diagram shows. A DaemonSet fluent-bit is meant to run one Pod per node. The DaemonSet controller watches the Node API resource set; for each node satisfying the template’s nodeSelector/nodeAffinity/tolerations constraints, it creates one Pod whose spec.affinity.nodeAffinity is set to pin it to that specific node by metadata.name — and then the default scheduler (kube-scheduler) binds it. This default-scheduler path is the modern norm: the ScheduleDaemonSetPods feature gate landed alpha in K8s 1.11 (PR #59862), was promoted to beta and turned on by default in 1.12 (issue #66526), reached GA in 1.17 (PR #82795), and the gate itself was removed in 1.18 — so for any cluster on 1.12 or later the scheduler binds DaemonSet Pods. Before 1.12 the controller set spec.nodeName directly, bypassing the scheduler. Node node-2 has a dedicated=gpu:NoSchedule taint — to land on it, the DaemonSet Pod needs a matching toleration in its template. Node node-3 is NotReady and has MemoryPressure; ordinary Pods would be evicted, but DaemonSet Pods inherit node.kubernetes.io/not-ready:NoExecute and node.kubernetes.io/memory-pressure:NoSchedule tolerations automatically from the controller — so the log shipper keeps running. When node-4 joins, the controller observes the new Node object and creates a Pod for it; when a node is deleted, the orphaned Pod is garbage-collected via owner references. The insight to extract is that DaemonSet is the only built-in workload that ties replica count to fleet size automatically, and the only one whose Pods are pre-configured to survive node-condition taints — both consequences of the “infrastructure Pod” mental model the resource is designed around.

Mechanical Walk-through

The controller’s reconcile loop

The DaemonSet controller watches three resource types:

  1. DaemonSets — its primary resource.
  2. Nodes — to react to node joins, deletions, label changes, and taint changes.
  3. Pods owned by DaemonSets — to detect failures and trigger replacement.

On reconcile, for each DaemonSet, the controller computes the set of eligible nodes by:

  • Filtering the cluster’s Nodes against spec.template.spec.nodeSelector (if set).
  • Filtering further against spec.template.spec.affinity.nodeAffinity (if set).
  • For each candidate node, checking whether the Pod’s tolerations (template tolerations + the auto-added defaults) cover the node’s taints. Nodes whose taints aren’t tolerated are excluded.

For each eligible node without an existing matching Pod, the controller creates one — with spec.affinity.nodeAffinity patched to pin to that node by name. The default scheduler then performs the actual binding (it sets .spec.nodeName on the Pod after running its predicates against the node affinity term the controller injected — see kubernetes.io — how-daemon-pods-are-scheduled). This delegation is controlled by the ScheduleDaemonSetPods feature gate, which had a multi-release lifecycle worth pinning precisely: alpha in 1.11, default-on beta in 1.12, GA in 1.17, gate removed in 1.18 (per PR #59862, issue #66526, PR #82795, and the removed-feature-gates reference). Practically, any supported K8s cluster (1.27+ at time of writing) uses the default-scheduler path; the older controller-sets-nodeName mechanism is a piece of pre-1.12 history. The consequence is that DaemonSet Pods participate in normal scheduling features — priority-and-preemption, topology spread, scheduling framework plugins — that the bypassed-scheduler path did not honor.

The auto-injected tolerations

The DaemonSet controller automatically adds the following tolerations to every DaemonSet Pod template (kubernetes.io — taints-and-tolerations; cross-checked against the documentation page snapshot):

TolerationEffectRationale
node.kubernetes.io/not-readyNoExecutePod survives a node-not-ready condition; don’t evict the log shipper when the node is having trouble.
node.kubernetes.io/unreachableNoExecuteSurvives a node-unreachable condition (kubelet can’t be reached by the control plane).
node.kubernetes.io/disk-pressureNoScheduleCan land on a disk-pressured node (the node-exporter must still run).
node.kubernetes.io/memory-pressureNoScheduleCan land on a memory-pressured node.
node.kubernetes.io/pid-pressureNoScheduleCan land on a PID-pressured node.
node.kubernetes.io/unschedulableNoScheduleCan land on a cordoned node — critical, because a cordoned-for-maintenance node still needs its CNI and logs to keep functioning.
node.kubernetes.io/network-unavailableNoScheduleOnly added when the Pod template requests hostNetwork: true, per the docs. Lets CNI bootstrap Pods land on nodes that haven’t yet had networking configured (the chicken-and-egg of CNI installation).

The seven-toleration list above is reproduced verbatim from the kubernetes.io DaemonSet page’s Taints and tolerations section as of K8s 1.30 docs (kubernetes.io — taints-and-tolerations) — these are the tolerations the DaemonSet controller injects, separate from the Pod-level eviction tolerations that the kubelet/admission controllers manage for ordinary Pods. The user’s template tolerations are additive to these — explicitly tolerating node-role.kubernetes.io/control-plane:NoSchedule is the standard way to also run a DaemonSet on the control-plane nodes (the default behavior excludes them on most clusters). To inspect the final tolerations actually applied to a specific Pod (template + auto-injected), run kubectl describe pod <daemonset-pod> and read the Tolerations: line.

Update strategies

Two (kubernetes.io — updating-a-daemonset):

  • RollingUpdate (default since K8s 1.6). On spec.template change, the controller replaces Pods one at a time. Two knobs:
    • maxUnavailable (default 1). The maximum number of DaemonSet Pods that can be unavailable during the rollout. Can be an integer or a percentage of the total eligible-node count.
    • maxSurge (GA in K8s 1.25 per kubernetes.io blog 2022-09-15). The maximum number of Pods above one-per-node that can transiently exist during the rollout — meaning the new Pod is created and brought to Ready before the old Pod is killed. Default 0 (no surge; classic kill-then-create). Cannot both be 0 simultaneously. maxSurge is incompatible with hostPort because two co-located Pods can’t bind the same node port simultaneously.
  • OnDelete. The controller does not update Pods on template change. Users (or a separate operator) must delete Pods to trigger replacement with the new template.

The combination is significant: with maxSurge: 1, maxUnavailable: 0, you get a true zero-downtime per-node rollout for DaemonSets that don’t bind host ports — the new Pod is created and verified Ready before the old one is killed. This is the modern best practice for log shippers, monitoring agents, and other DaemonSets that don’t require port exclusivity.

Node joins, leaves, label changes

  • Node joins. The DaemonSet controller reacts to the watch event for a new Node, evaluates eligibility, creates a Pod. The pace at which new nodes get their DaemonSet Pods is bounded by the controller’s reconcile rate plus the scheduler’s binding rate; in practice <5 seconds.
  • Node leaves. When a Node object is deleted, the orphaned Pods are removed by the garbage collector via owner references (Node ownership of Pods is implicit — the Pod isn’t really owned by the Node, but kubelet drops the Pod when the node leaves the cluster). For nodes that are unreachable rather than deleted, the not-ready:NoExecute toleration on DaemonSet Pods prevents eviction (which is desired — see above).
  • Node label changes. If a node’s labels change such that it no longer matches the DaemonSet’s nodeSelector, the controller deletes the DaemonSet Pod from that node. If a node gains the matching labels, a Pod is created.
  • Node taint changes. Same logic: gaining a tolerated taint changes nothing; gaining an untolerated taint causes the controller to delete the DaemonSet Pod (via the NoExecute effect, or implicitly by the next reconcile noticing ineligibility).

Why not just run a replicas=N Deployment with topology spread?

A common question: “Why not just deploy a regular Deployment with replicas: nodeCount and a topology spread constraint that forces one-per-node?” The answer:

  1. Replica count auto-tracking. A DaemonSet’s “replica count” is the eligible-node count. As the cluster scales from 50 to 200 nodes, the DaemonSet automatically grows to 200 Pods. A Deployment requires explicit reconfiguration (or external automation like KEDA) to track this.
  2. Default tolerations. Deployment Pods do not get the DaemonSet’s automatic tolerations for node-condition taints. A log-shipper Deployment Pod would get evicted from a memory-pressured node — exactly when you most need the logs of what is causing the memory pressure.
  3. Per-node identity (informal). A DaemonSet’s Pod-per-node maps cleanly to “this Pod represents this node’s role.” Tools downstream (kube-state-metrics, Prometheus pod-to-node joins, log indexing by node) rely on this 1:1 relationship.
  4. Eviction priority. DaemonSet Pods are typically high-priority via PriorityClass and are not evicted by node-pressure eviction the same way (the kubelet eviction logic special-cases certain DaemonSet workloads via QoS Classes + priority). Replicating this on a Deployment requires explicit PriorityClass configuration.

In short, DaemonSet exists because the abstraction “one Pod per node, with infrastructure-like resilience” is genuinely different from “N replicas spread evenly.”

Configuration / API Surface

A canonical Fluent Bit log shipper with line-by-line commentary:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit                       # immutable; must match template labels
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1                     # at most 1 Pod down at a time during rollout
      maxSurge: 0                           # default; safe for hostPort users (none here, but illustrative)
  minReadySeconds: 10                       # 10s post-Ready stabilization
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit        # for reading node-level resources / writing to S3
      priorityClassName: system-node-critical # so kubelet doesn't evict on pressure
      hostNetwork: false                    # set true for CNI/kube-proxy/host-port use cases
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule                # opt-in: also run on control-plane nodes
        - key: dedicated
          operator: Exists
          effect: NoSchedule                # tolerate "dedicated GPU node" taint
      containers:
        - name: fluent-bit
          image: cr.fluentbit.io/fluent/fluent-bit:3.0
          resources:
            requests: { cpu: "50m",  memory: "64Mi" }
            limits:   { cpu: "200m", memory: "128Mi" }
          volumeMounts:
            - name: varlog                  # node log directory
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers  # container log directory (containerd uses /var/log/containers)
              mountPath: /var/lib/docker/containers
              readOnly: true
          livenessProbe:
            httpGet: { path: /, port: 2020 }
            periodSeconds: 30
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log                  # raw host path — DaemonSet's defining capability
            type: Directory
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
            type: DirectoryOrCreate

Notable surface points:

  • hostPath volumes are the DaemonSet’s defining capability — to monitor or ship logs from a node, you need access to its filesystem. Use readOnly: true and a narrow path where possible; broad / mounts are an exfiltration risk.
  • hostNetwork: true is common for CNI agents, kube-proxy, and other components that need to manipulate the host’s network. With hostNetwork: true, the Pod uses the node’s IP and ports directly (no Pod IP allocation); port conflicts between DaemonSet Pods and node services are then possible.
  • priorityClassName: system-node-critical is the canonical priority class for infrastructure DaemonSets — protects the Pod from eviction. The built-in classes system-cluster-critical and system-node-critical come with K8s; system-node-critical is the higher of the two.
  • updateStrategy.rollingUpdate.maxSurge: 1 (with maxUnavailable: 0) is the zero-downtime modern rollout — confirmed GA in 1.25. Incompatible with hostPort; safe for most modern observability agents that don’t bind host ports.

Failure Modes

  1. Pod can’t land on the control plane. The user’s DaemonSet template lacks a node-role.kubernetes.io/control-plane:NoSchedule toleration, so the Pod skips the control-plane nodes. Surprising for log shippers and monitoring — the user assumes coverage is universal. Diagnose: kubectl get pods -o wide shows no DaemonSet Pod on control-plane node names. Fix: add the toleration.

  2. maxSurge + hostPort. Setting maxSurge: 1 on a DaemonSet that uses hostPort: 9100 (e.g., node-exporter) causes the new Pod to fail to bind because the old Pod still owns the port. The rollout deadlocks. Fix: use maxUnavailable for hostPort users, not maxSurge. GitHub kubernetes#111194 documents this incompatibility explicitly.

  3. hostPath security incident. A DaemonSet with / mounted readable lets any tenant in the cluster’s logging namespace read every node’s filesystem — including kubelet credentials and other tenants’ container filesystems. Symptom: an internal security audit flags it; no functional failure until exploited. Fix: scope the hostPath narrowly; combine with Pod Security Standards (Restricted profile bans hostPath).

  4. DaemonSet Pod consuming the whole node. With no resources.limits, a buggy log shipper that leaks memory can OOM the entire node — including critical workload Pods. Symptom: node goes NotReady, kubelet OOM-kill events in kubectl describe node. Fix: always set conservative limits on DaemonSet Pods; they’re the most dangerous in this regard because they run everywhere.

  5. Stale Pods on cordoned nodes. When a node is cordoned for maintenance, node.kubernetes.io/unschedulable:NoSchedule is added — but DaemonSet Pods tolerate it (auto-toleration). This is desired for some DaemonSets (the CNI must keep working until the node leaves) but undesired for others (the log shipper accumulating un-shipped logs as the node drains). Operators sometimes explicitly un-tolerate the unschedulable taint to opt into draining. Diagnose: kubectl get pods -o wide showing DaemonSet Pods still on cordoned nodes is the normal outcome.

  6. Template change pushing the wrong image to CNI nodes during upgrade. Bumping the Cilium DaemonSet’s image to a version incompatible with the kubelet version on the node causes CNI to fail to bring up new Pods. Since the new CNI Pod fails Ready, the rolling update halts at the first node — but in the meantime, that node has a degraded CNI dataplane. Diagnose: kubectl get pods -l app=cilium shows the first updated Pod CrashLoopBackOff. Fix: revert via kubectl rollout undo.

  7. DaemonSet rollout interacting with Cluster Autoscaler scale-down. The CA scales down underutilized nodes by draining them. During the drain, the DaemonSet Pod tolerating unschedulable keeps running — but CA waits for all non-DaemonSet Pods to terminate, then deletes the node. If a DaemonSet Pod has a terminationGracePeriodSeconds: 600, the node deletion appears to hang for 10 minutes. Diagnose: nodes lingering in NotReady state during scale-down. Fix: lower grace period on DaemonSets where appropriate.

  8. OnDelete strategy + forgotten updates. Switching to OnDelete for a CNI DaemonSet (so operators can manually orchestrate updates) and then forgetting that template changes don’t auto-roll. A security patch to the CNI image sits unapplied for months. Diagnose: kubectl get pods -o jsonpath showing old image tags despite spec being updated. Fix: explicit operator process to delete Pods after template change.

  9. DaemonSet count != node count due to taints. A 100-node cluster shows 80 fluent-bit Pods — confusing until you realize 20 nodes have a tolerable-but-untolerated taint. Diagnose: kubectl describe nodes listing all taints; cross-reference with DaemonSet template tolerations.

  10. kube-proxy as DaemonSet vs static Pod. On most managed K8s, kube-proxy runs as a DaemonSet; on some self-managed clusters, it’s a static Pod. The two have different rollout semantics — DaemonSet uses RollingUpdate/OnDelete; static Pods are kubelet-managed and update on file change. Mixing the assumption causes surprise during cluster upgrades.

Alternatives and When to Choose Them

  • Deployment + topology spread constraint — choose when you want approximately one-per-node but explicit replica control matters more than auto-tracking node count. Loses default tolerations; gains explicit knobs.
  • Static Pods — choose when the workload must come up before the API server is reachable (kube-apiserver itself, etcd, kube-controller-manager on bootstrap nodes). The kubelet reads pod manifests from a local directory (/etc/kubernetes/manifests) and runs them without API server involvement. Used by kubeadm to bootstrap the control plane. See Static Pods.
  • Per-node Init Pods via Job — choose when the workload is run-once-per-node-then-exit rather than long-running (e.g., one-time TLS cert generation on each node). A DaemonSet with a long-running pause container that does the work and sleeps is an anti-pattern; use a Job with node-targeting (one Job per node, or a per-node node-feature-discovery hook).
  • Operator with custom node-controller — choose when the per-node Pod needs orchestration beyond what DaemonSet offers (rolling restarts coordinated with external state, per-node configuration with finalizers, etc.). The NVIDIA GPU Operator and the Cilium operator both layer custom controllers on top of DaemonSets for this reason.
  • System packages on the node OS — sometimes the right answer is not to run it on K8s at all. The node’s logging agent could be installed via OS package and managed by Ansible/Chef. K8s-as-a-platform purists hate this; pragmatic SREs use it where the agent predates the cluster or where OS-level integration is required.

Production Notes

  • Cilium and Calico both ship as DaemonSets. The Cilium agent DaemonSet plus the cilium-operator (a Deployment) is the canonical install pattern; the agent handles per-node BPF/datapath programming. Calico’s calico-node DaemonSet hosts Felix (the per-node enforcer) and BIRD (the BGP daemon). See Cilium / Calico.
  • kube-proxy on most managed K8s is a DaemonSet (EKS, GKE-standard, AKS). On GKE Autopilot it’s hidden; on OpenShift it’s wrapped in OVN-Kubernetes’ DaemonSet. The fact that even kube-proxy — a core K8s component — is shipped as a DaemonSet illustrates how foundational the abstraction is.
  • CSI node plugins as DaemonSets. Every CSI driver has a “node plugin” DaemonSet that runs on each node and handles mount/unmount; the “controller plugin” is typically a separate Deployment. The AWS EBS CSI driver’s ebs-csi-node DaemonSet is the canonical example.
  • Spotify’s logging pipeline uses a Fluent Bit DaemonSet per node forwarding to a centralized Kafka cluster, then to BigQuery for indexing. The DaemonSet’s resilience to node-condition taints is critical — Spotify explicitly cites cases where the log shipper kept running on memory-pressured nodes long enough to ship the OOM-event logs that explained the pressure.
  • k8s.af incidents involving DaemonSets (Kubernetes Failure Stories) most commonly involve: (a) DaemonSets without resource limits OOM-ing nodes; (b) CNI DaemonSets bricking nodes during a botched upgrade (no networking → can’t pull new image → stuck); (c) hostPath mounts becoming security incidents; (d) updates that didn’t propagate because OnDelete was set and the team forgot.
  • Karpenter scale-down + DaemonSet. Karpenter (Karpenter) explicitly accounts for DaemonSet Pod resource requests when bin-packing — a node hosting 200m + 256Mi worth of DaemonSet Pods is allocated proportionally less for workloads. Without this accounting, clusters with many DaemonSets (CNI + CSI + logger + monitor + security agent = 5 DaemonSets, easily 500m CPU per node) end up over-provisioned for workload Pods that can’t actually fit.

See Also

Graph View

Backlinks