vault

Static Pods

16 min read

Static Pods

A static Pod is a Pod managed directly by a single node’s kubelet, without involvement from the Kubernetes API server. kubelet watches a configured directory (default /etc/kubernetes/manifests/) — or, less commonly, a URL — for Pod-manifest YAML files; for each file it finds, it runs the Pod on the local node, restarts it on failure, and publishes a read-only representation of the Pod on the API server called a “mirror Pod” so that kubectl get pods can see it (Kubernetes — Create static Pods). Static Pods exist primarily to solve the control-plane bootstrap chicken-and-egg problem: kubeadm-installed clusters run kube-apiserver, etcd, kube-scheduler, and kube-controller-manager as static Pods on each control-plane node, so the apiserver does not need to be running in order for the apiserver to start (Kubernetes — Bootstrapping with kubeadm). Outside the bootstrap use case, static Pods are tightly limited — no rolling updates, no ResourceQuotas, no admission controllers, no ServiceAccounts, no ConfigMap/Secret references — which is why DaemonSets, not static Pods, are the correct tool for node-level agents in normal operation.

Mental Model

flowchart LR
    subgraph "Control-plane Node (e.g., master-0)"
        MANIFEST_DIR["/etc/kubernetes/manifests/<br/>kube-apiserver.yaml<br/>etcd.yaml<br/>kube-scheduler.yaml<br/>kube-controller-manager.yaml"]
        KUBELET["kubelet<br/>--config /var/lib/kubelet/config.yaml<br/>(staticPodPath: /etc/kubernetes/manifests)"]
        CRI["CRI runtime<br/>(containerd / CRI-O)"]
        POD_APISERVER["Pod: kube-apiserver<br/>(actually running)"]
        POD_ETCD["Pod: etcd<br/>(actually running)"]

        MANIFEST_DIR -->|"periodic scan (fileCheckFrequency, 20s)"| KUBELET
        KUBELET -->|"RunPodSandbox + CreateContainer"| CRI
        CRI --> POD_APISERVER
        CRI --> POD_ETCD
    end

    subgraph "API Server (running inside the static Pod above)"
        APISERVER["kube-apiserver"]
        ETCD[("etcd")]
        APISERVER <--> ETCD
    end

    POD_APISERVER -.->|"the static Pod IS the apiserver"| APISERVER
    KUBELET -->|"POST /api/v1/pods (mirror)"| APISERVER
    APISERVER -->|"mirror Pod object visible via kubectl"| OPERATOR

    OPERATOR["kubectl get pods -n kube-system"] -->|"reads mirror pods"| APISERVER

    classDef static fill:#fde,stroke:#a44
    class POD_APISERVER,POD_ETCD,APISERVER,ETCD static

Caption. The diagram is deliberately recursive at the highlighted node. The kube-apiserver process (right box) is running inside the static Pod kubelet started from /etc/kubernetes/manifests/kube-apiserver.yaml. Once the apiserver is up, kubelet POSTs a mirror Pod representation back to it so operators can see what’s running. The arrow from kubelet to apiserver demonstrates the temporal independence: kubelet started the static Pod before the apiserver was reachable. If you cordoned this node and the apiserver crashed, kubelet would still restart it from the manifest on disk — that’s the whole point. The insight: static Pods break circular dependencies by giving kubelet a manifest source that is not the apiserver itself.

Mechanical Walk-through

What kubelet Does With the Manifest Directory

kubelet’s configuration (typically /var/lib/kubelet/config.yaml) contains a staticPodPath field — historically also settable via the deprecated CLI flag --pod-manifest-path — pointing at the manifest directory. When set, kubelet:

  1. Walks the directory at startup and loads every file that doesn’t begin with a dot (the leading-dot rule lets you keep .bak files in the same directory without them being interpreted as Pod manifests — a real footgun that the docs explicitly warn about).
  2. Periodically re-reads the directory to pick up changes. The official documentation describes this plainly as the kubelet “periodically scan[ning]” the directory and adding/removing Pods as files appear or disappear (Kubernetes — Create static Pods). The scan cadence is governed by the kubelet’s fileCheckFrequency setting, default 20s (Kubelet Configuration v1beta1) — which is why a manifest change or removal can take up to ~20 seconds to take effect rather than being instantaneous. (Internally the kubelet’s file config-source does also register a filesystem watch via fsnotify so prompt edits are noticed, but the fileCheckFrequency resync is the contract the docs commit to and the upper bound on propagation; do not rely on sub-second reaction.)
  3. Parses each file as a Pod spec. Files that fail to parse are skipped with a kubelet log entry and the file is otherwise ignored — there is no admission controller, no validation webhook, no quota check.
  4. Creates each Pod through the same CRI calls as any other Pod — RunPodSandbox to bring up the Pause Container, then CreateContainer and StartContainer for each user container. From the container runtime’s perspective, a static Pod is indistinguishable from an API-server-managed Pod.
  5. Names each static Pod by appending -<node-name> to its metadata.name. This is why kubectl get pods -n kube-system on a kubeadm cluster shows kube-apiserver-master-0, kube-apiserver-master-1, etc.
  6. Posts a mirror Pod. kubelet performs an authenticated POST to /api/v1/namespaces/<ns>/pods carrying the Pod spec plus an annotation kubernetes.io/config.mirror: <hash> and kubernetes.io/config.source: file. The mirror Pod’s nodeName is set to this node; its ownerReferences link it back to the kubelet that created it.
  7. Treats the manifest as the source of truth. If a user deletes the mirror Pod via kubectl delete, kubelet will recreate it within seconds because the on-disk manifest is unchanged. The only ways to remove a static Pod are to delete the manifest file or to move it out of the directory.

The reverse: edit the manifest file on disk and, on its next scan (within fileCheckFrequency, default 20s), kubelet detects the change, performs a “config update” reconcile, and recreates the Pod with the new spec (no rolling update — straight kill + restart). Move the file out of the directory and kubelet stops the Pod and removes the mirror.

Web-Hosted Manifests (Less Common)

An alternative source: kubelet can be pointed at an HTTP URL that returns a Pod manifest, configured via the kubelet-config field staticPodURL (historically the deprecated CLI flag --manifest-url), which it refetches on a schedule. Used historically in early CoreOS configurations and some IoT-style fleets where a central server pushes manifests; in 2026 it is essentially unused in favor of file-based manifests plus automation-based configuration management (Ansible, Salt, cloud-init). The docs still describe it but the file-directory path is the universal default.

Mirror Pods — What They Are and Are Not

A mirror Pod is a read-only API object. The annotation kubernetes.io/config.source: file (or http) marks it. Important non-properties:

  • Cannot be edited via kubectl edit — any edit is reverted at the next kubelet reconcile.
  • Cannot reference ServiceAccounts, ConfigMaps, or Secrets via the normal Pod-spec fields. The kubelet cannot resolve API-managed references when it might be running before the apiserver. Files mounted via hostPath are the only configuration mechanism static Pods support cleanly. ServiceAccount tokens that are present (kube-apiserver itself needs the cluster signing key) get there by being on disk in /etc/kubernetes/pki/ and hostPath-mounted into the Pod.
  • No admission control runs. ValidatingAdmissionWebhook, MutatingAdmissionWebhook, OPA Gatekeeper, Kyverno — none of them see static Pods because the Pod is created by kubelet, not via an API POST. Sidecar injection mutating webhooks (Istio’s, Linkerd’s) do not apply.
  • No ResourceQuota enforcement. The manifest’s requests/limits are honored by the scheduler’s node-local logic (kubelet won’t schedule a static Pod that doesn’t fit on the node), but quota objects are ignored.
  • No PodDisruptionBudget. kubectl drain will not delete static Pods; you must remove the manifest by hand or use --force --ignore-daemonsets --delete-emptydir-data and accept that drain doesn’t actually evict them.
  • No Deployment-style features. No rolling update, no rollback, no kubectl rollout status. Two static Pod manifests for “the same component” are two independent Pods.
  • No ephemeral containers (kubectl debug against a static Pod does not work).
  • No PriorityClass-based scheduling pressure — they cannot preempt other Pods because they’re scheduled by kubelet directly.

The Bootstrap Sequence — kubeadm in Detail

When kubeadm init runs on a fresh control-plane node, the sequence is roughly:

  1. Pre-flight checks (kernel modules, swap off, container runtime present, etc.).
  2. Certificate generation under /etc/kubernetes/pki/ (ca.crt, ca.key, apiserver.crt, apiserver-kubelet-client.crt, etcd certs, sa.key for service-account signing).
  3. kubeconfig generation for kubelet, controller-manager, scheduler, admin.
  4. Static Pod manifest generation for etcd, kube-apiserver, kube-controller-manager, kube-scheduler under /etc/kubernetes/manifests/.
  5. Start kubelet. kubelet wakes, reads its config (which points staticPodPath at /etc/kubernetes/manifests/), reads the four manifests, and asks the container runtime to start them.
  6. etcd comes up first (it’s a self-contained binary; nothing else depends on it being healthy yet).
  7. kube-apiserver starts, connects to etcd, and serves the API.
  8. kube-controller-manager and kube-scheduler start, register with the apiserver, begin their reconcile loops.
  9. kubelet posts mirror Pods. Now kubectl get pods -n kube-system shows etcd-<node>, kube-apiserver-<node>, kube-controller-manager-<node>, kube-scheduler-<node>.
  10. kubeadm completes with the well-known “join token” output for adding worker nodes.

Worker-node kubeadm join does not create static Pods; worker nodes have no /etc/kubernetes/manifests/ directory in use. Their kubelet talks straight to the apiserver and runs only API-managed Pods (DaemonSets for CNI agents, kube-proxy, etc.).

If you upgrade a control plane in place, kubeadm upgrade swaps the manifest files atomically (write to a tmp file, then rename into place — so the kubelet never observes a half-written manifest on its next scan) so kubelet sees a single clean transition and restarts the Pod. This is the closest thing static Pods have to a rolling update, and it operates one node at a time.

Configuration / API Surface

A real static-Pod manifest (a stripped kube-apiserver manifest as kubeadm would generate it):

# /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver           # kubelet appends -<node-name> for the mirror
  namespace: kube-system
  labels:
    component: kube-apiserver
    tier: control-plane
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.0.0.10:6443
spec:
  hostNetwork: true               # use the host's network namespace directly
  priorityClassName: system-node-critical
  containers:
    - name: kube-apiserver
      image: registry.k8s.io/kube-apiserver:v1.36.0   # illustrative; kubeadm pins the cluster's exact patch version
      imagePullPolicy: IfNotPresent
      command:
        - kube-apiserver
        - --advertise-address=10.0.0.10
        - --allow-privileged=true
        - --authorization-mode=Node,RBAC
        - --client-ca-file=/etc/kubernetes/pki/ca.crt
        - --enable-admission-plugins=NodeRestriction
        - --enable-bootstrap-token-auth=true
        - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
        - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
        - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
        - --etcd-servers=https://127.0.0.1:2379
        - --service-account-issuer=https://kubernetes.default.svc.cluster.local
        - --service-account-key-file=/etc/kubernetes/pki/sa.pub
        - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
        - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
        - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
      livenessProbe:
        httpGet: { host: 127.0.0.1, path: /livez, port: 6443, scheme: HTTPS }
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 15
      resources:
        requests: { cpu: 250m }
      volumeMounts:
        - name: ca-certs
          mountPath: /etc/ssl/certs
          readOnly: true
        - name: k8s-certs
          mountPath: /etc/kubernetes/pki
          readOnly: true
  volumes:
    - name: ca-certs
      hostPath: { path: /etc/ssl/certs, type: DirectoryOrCreate }
    - name: k8s-certs
      hostPath: { path: /etc/kubernetes/pki, type: DirectoryOrCreate }

Line-by-line commentary.

  • No ServiceAccount, ConfigMap, or Secret references anywhere. All configuration is via command-line flags and hostPath mounts of files on the node’s local filesystem. The apiserver doesn’t exist yet, so it cannot mount its own Secrets.
  • hostNetwork: true means the apiserver listens on the node’s own IP rather than getting a Pod IP from CNI. This matters because CNI itself is a DaemonSet (managed via apiserver) — the apiserver couldn’t bring up CNI for its own networking.
  • priorityClassName: system-node-critical — even though PriorityClass enforcement is via the scheduler (which doesn’t schedule static Pods), kubelet uses this to influence node-pressure eviction; the apiserver Pod is essentially un-evictable.
  • command uses long-form --flag=value arguments rather than the args/env pattern most application Pods use, because the apiserver’s configuration surface is its CLI.
  • The livenessProbe uses HTTPS on 127.0.0.1:6443 — the apiserver checks its own health from the same Pod’s network namespace (which is the host’s, given hostNetwork: true).

To create a non-control-plane static Pod on a node (for example, to test):

$ sudo mkdir -p /etc/kubernetes/manifests
$ cat <<EOF | sudo tee /etc/kubernetes/manifests/static-nginx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-nginx
spec:
  containers:
    - name: web
      image: nginx:1.25
      ports:
        - containerPort: 80
EOF
# Within a few seconds:
$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
static-nginx-worker-3     1/1     Running   0          12s
# kubectl cannot manage it:
$ kubectl delete pod static-nginx-worker-3
pod "static-nginx-worker-3" deleted
$ sleep 5; kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
static-nginx-worker-3     1/1     Running   0          4s    # ← came back
# Only removing the manifest stops it:
$ sudo rm /etc/kubernetes/manifests/static-nginx.yaml
$ sleep 5; kubectl get pods
No resources found in default namespace.

This experiment is the canonical “ah-ha” demonstration of what “kubelet-managed, not API-managed” actually means.

Failure Modes

Manifest typos fail silently in odd ways. Because no admission controller runs, a manifest with an invalid spec (misspelled field, missing required field, image that doesn’t exist) produces a Pod that simply doesn’t start, with the failure recorded only in kubelet’s log on that one node — not as an admission rejection visible via kubectl describe. Diagnose with journalctl -u kubelet --since "5 minutes ago" | grep -i error on the node, or crictl ps -a to see Pods in CreateContainerError state. The lack of feedback is one of the most cited static-Pod pain points.

Backup files in the manifest directory become live Pods. A common operator mistake: cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak to keep a backup before editing. Because .bak does not start with a dot, kubelet parses it as a Pod manifest, encounters the same Pod name as the live apiserver, and chaos ensues. The docs explicitly tell you to keep backups outside the directory; the rule is “files NOT starting with a dot are loaded.” A safer convention is mv to a sibling backup directory like /etc/kubernetes/manifests-backup/.

Editing a control-plane manifest crashes the control plane until kubelet reloads. When the kubelet’s next scan picks up the modified file; if the new manifest is invalid, the existing Pod is killed and the replacement fails to start, leaving the apiserver / scheduler / controller-manager down on that node. On HA control planes this is per-node degradation; on single-node control planes (or when you’re rolling all three at once) this is total cluster downtime until the file is corrected. The mitigation is to always validate the new manifest with kubectl --dry-run=client -f or kubeadm config print before placing it, and to do one node at a time.

Mirror-pod orphans on node deletion. If a node is forcibly deleted from the cluster while its kubelet is offline, its mirror Pods linger in etcd. They show up in kubectl get pods -n kube-system but have no live kubelet behind them. The Node controller’s garbage collection eventually removes them, but the lag can confuse troubleshooting.

Static Pod resource accounting consumes ResourceQuota. A mirror Pod is a real Pod object in its namespace, and the ResourceQuota pod evaluator does not special-case it. The quota evaluator’s QuotaV1Pod filter excludes only Pods in a terminal phase (Failed/Succeeded) and Pods already past their deletion grace period; it has no check for the kubernetes.io/config.mirror / kubernetes.io/config.source annotations (kubernetes source — pkg/quota/v1/evaluator/core/pods.go). So a running mirror Pod does count toward pods, requests.cpu, requests.memory, etc. in a quota-controlled namespace. In practice the canonical control-plane static Pods sit in kube-system, which is rarely quota-controlled, so this seldom bites — but placing application static Pods in a namespace that is quota-controlled silently consumes that namespace’s budget, and (because there is no admission path for static Pods) the quota cannot reject them the way it would a normal kubectl apply over budget; it simply records the usage, which can then block subsequent API-managed Pods. The operational guidance: keep static Pods out of quota-controlled namespaces.

No SecurityContext / Pod Security Admission enforcement. Pod Security Admission is an admission controller; it does not run for static Pods. A static Pod manifest with hostNetwork: true, privileged containers, or host-path mounts of /etc will run with full host access regardless of the namespace’s PSA label. This is fine for control-plane Pods (which need exactly this) and unsafe for anything else, which is one more reason application workloads should not be static Pods.

Alternatives and When to Choose Them

For node-level agents that must run on every node — use a DaemonSet. A DaemonSet (Kubernetes — DaemonSet) is API-managed, supports rolling updates, ResourceQuotas, ConfigMaps, ServiceAccounts, Tolerations, admission policies, and works with kubectl drain. Logging agents (Fluent Bit, Vector), monitoring agents (Prometheus node-exporter), CNI plugins (Calico, Cilium), and kube-proxy itself are DaemonSets. Use static Pods only when the workload must start before the apiserver — which in practice means only the apiserver, etcd, controller-manager, and scheduler themselves.

For control-plane components in self-hosted clusters — alternatives to static Pods exist. Cluster API and similar projects (Kamaji, k0s, k3s in some modes) run the control plane as ordinary API-managed Pods in a separate “management” cluster, breaking the bootstrap loop by having a different cluster’s apiserver manage them. This is operationally cleaner for managed-K8s services but adds the dependency on a meta-cluster.

For “I want a Pod that survives apiserver outages” — that’s not actually a thing static Pods give you. Any Pod that’s already running keeps running through an apiserver outage; kubelet’s cached nodeStatus and per-Pod containers don’t lose state. The difference for static Pods is that new instances can be created during the outage, because kubelet doesn’t need apiserver to start them. For typical applications this is irrelevant — they wouldn’t be scheduling new replicas mid-outage anyway.

For local development clusters — minikube, kind, and k3d use static Pods internally for the control plane the same way kubeadm does. Operators rarely see them, but the mechanism is identical.

Production Notes

kubeadm and the static-Pod control plane is the industry default. Every kubeadm-installed cluster, every Cluster API workload cluster (by default), every on-prem install runs control-plane components as static Pods. The pattern is so universal that “control plane node” and “node with /etc/kubernetes/manifests/ populated” are operationally equivalent on most distributions. The exceptions are managed-Kubernetes services (EKS, GKE, AKS) where the cloud provider runs the control plane outside the worker fleet entirely.

kubeadm upgrade apply is, mechanically, a static-Pod manifest update. When you upgrade a control-plane minor version, kubeadm computes new manifests for the four components, atomically renames them into place, and waits for kubelet to bring up the new Pods. The whole upgrade is a sequence of file writes plus health checks; understanding static Pods is what lets you debug a stuck upgrade.

Etcd backups and static Pods. The conventional way to back up etcd on a kubeadm cluster is etcdctl snapshot save invoked against the local etcd static Pod’s listener. Restoring is the inverse: bring up etcd from snapshot, then let the static-Pod kube-apiserver restart against the restored etcd. The whole DR play depends on the manifest files surviving — which is why /etc/kubernetes/ is part of every production backup.

Worker-node static Pods are an anti-pattern. Outside the control plane, static Pods on worker nodes show up in occasional “I need to run X on this specific node before kubelet talks to apiserver” use cases — typically log-shipper bootstrap or per-node disk preparation. The cleaner pattern is a DaemonSet with appropriate node selectors plus an init container; the use case for true worker static Pods is narrow enough that most production clusters have zero.

The Borg lineage. Borg had a similar concept of “Borgmaster bootstrap” where the master itself needed to come up before scheduling anything; the K8s static-Pod pattern is a direct descendant of that bootstrap-via-local-config approach (Burns et al. 2016). The “control plane runs on the same machinery as workloads” choice is unusual among orchestrators (Mesos and Nomad keep the masters separate from worker fleets) and is what makes the K8s control plane self-host-able.

See Also

Graph View

Backlinks