vault

Pod Eviction

13 min read

Pod Eviction

The word “eviction” in Kubernetes names two entirely unrelated mechanisms, and conflating them is one of the most common sources of operational confusion. API-initiated eviction is a voluntary, policy-respecting operation: a client (a kubectl drain, the Cluster Autoscaler, the Descheduler) asks the API server, via the pods/eviction subresource, to gracefully remove a Pod — and the request is blocked if it would violate a Pod Disruption Budget (kubernetes.io — API-initiated Eviction). Node-pressure eviction is something else completely: the kubelet on a node that is running out of memory, disk, or process IDs kills Pods to save the node itself — and it does not respect Pod Disruption Budgets, because a node about to fall over is not a situation in which availability guarantees can be honoured (kubernetes.io — Node-pressure Eviction). This note pulls the two apart and then covers node-pressure eviction in depth, since it is the one that surprises people.

Mental Model

flowchart TB
    subgraph API["API-initiated eviction — VOLUNTARY"]
        DRAIN["kubectl drain /<br/>Cluster Autoscaler /<br/>Descheduler"]
        EVAPI["POST .../pods/&lt;name&gt;/eviction<br/>(Eviction subresource)"]
        PDBCHK{"would this violate<br/>a PodDisruptionBudget?"}
        DRAIN --> EVAPI --> PDBCHK
        PDBCHK -->|no| GRACE["graceful delete:<br/>respects terminationGracePeriod"]
        PDBCHK -->|yes| R429["429 Too Many Requests<br/>— retry later"]
    end
    subgraph NODE["Node-pressure eviction — INVOLUNTARY"]
        SIG["kubelet eviction manager<br/>polls signals:<br/>memory.available, nodefs.available,<br/>imagefs.available, pid.available"]
        THRESH{"crossed soft or<br/>hard threshold?"}
        RECLAIM["reclaim node-level:<br/>GC dead containers,<br/>GC unused images"]
        ORDER["still pressured ⇒<br/>evict Pods in order:<br/>BestEffort → Burstable&gt;requests → Guaranteed"]
        SIG --> THRESH -->|yes| RECLAIM --> ORDER
        ORDER -. "PDBs NOT consulted" .-> ORDER
    end

What this diagram shows. The two mechanisms live in completely different parts of the system. API-initiated eviction is an API call — it goes through the kube-apiserver, which runs the disruption-budget check before allowing the Pod to be deleted; a denied request returns HTTP 429. Node-pressure eviction is a kubelet local decision — the kubelet’s eviction manager watches kernel/filesystem signals on its own node and, when a node is starving, kills Pods directly without asking the API server’s permission and without consulting any PodDisruptionBudget. The insight to extract: API-initiated eviction protects applications; node-pressure eviction protects the node. When the two goals conflict — a node is dying but a PDB says “keep N replicas” — the node wins, because a crashed node helps no one.

Mechanical Walk-through

API-initiated eviction — the voluntary path

The Eviction API is a subresource of Pod: a client creates an Eviction object (apiVersion: policy/v1, kind: Eviction) by POSTing to .../namespaces/<ns>/pods/<name>/eviction. The API server treats this as “please delete this Pod, but only if disruption policy allows.” It checks every Pod Disruption Budget whose selector matches the Pod:

  • If removing the Pod would still leave the PDB satisfied (minAvailable met, or maxUnavailable not exceeded), the API server proceeds: the Pod gets a deletion timestamp, the kubelet runs the normal graceful-termination sequence (preStop hook → SIGTERM → wait up to terminationGracePeriodSeconds → SIGKILL), and the EndpointSlice controller drops the Pod from Services. Response: 200 OK.
  • If removing the Pod would violate the PDB, the API server refuses with 429 Too Many Requests — “not allowed right now, retry later.” A misconfiguration (two PDBs both matching the Pod) yields 500.

kubectl drain is the canonical caller: it cordons the node (marks it unschedulable), then issues an Eviction for each Pod, retrying on 429 until every evictable Pod is gone. The Cluster Autoscaler uses the same API to drain a node it wants to scale down; the Descheduler uses it to relocate mis-placed Pods. All of these are voluntary disruptions — planned, policy-bounded, PDB-respecting. This is the eviction you want.

Node-pressure eviction — the involuntary path

The kubelet’s eviction manager runs on a timer (10 s by default) and samples a set of eviction signals:

  • memory.available — node allocatable memory − working set (working set is computed from cgroup stats and excludes inactive_file, on the assumption that page cache is reclaimable).
  • nodefs.available / nodefs.inodesFree — free space / inodes on the node filesystem (/var/lib/kubelet: logs, emptyDir, ephemeral storage).
  • imagefs.available / imagefs.inodesFree — free space / inodes on the container runtime’s image filesystem.
  • containerfs.available / containerfs.inodesFree — the container writable-layer filesystem (a newer, separate-disk-GC signal).
  • pid.available — free process IDs on the node.

Each signal can have two thresholds:

  • A hard threshold (evictionHard) triggers eviction immediately, with zero grace — the kubelet SIGKILLs the victim Pod’s containers at once. Defaults: memory.available<100Mi, nodefs.available<10%, nodefs.inodesFree<5%, imagefs.available<15%, imagefs.inodesFree<5%, pid.available<....
  • A soft threshold (evictionSoft) triggers eviction only after the signal has stayed crossed for evictionSoftGracePeriod, and the evicted Pod gets a graceful termination capped by evictionMaxPodGracePeriod. Soft thresholds are for catching slow degradation before it becomes a hard breach.

When a threshold is crossed the kubelet does three things in order. First, it sets a node condition and applies a taint — MemoryPressure (node.kubernetes.io/memory-pressure), DiskPressure (node.kubernetes.io/disk-pressure), or PIDPressure (node.kubernetes.io/pid-pressure) — which stops the scheduler placing new Pods there (see Node Pressure Conditions). Second, it attempts node-level reclaim that does not kill user Pods: garbage-collecting dead containers and (for image-filesystem pressure) unused container images. Third, if pressure persists after reclaim, it evicts Pods.

The eviction ordering

When the kubelet must evict, it ranks Pods and kills the worst candidate first. The ranking is driven by QoS class and usage relative to request:

  1. BestEffort Pods first — Pods with no requests and no limits. They promised the scheduler nothing, so they are sacrificed first. Among BestEffort Pods, the one using the most of the pressured resource goes first.
  2. Burstable Pods exceeding their requests, next — Pods that have requests but whose actual usage is above what they requested. They are ranked by how far over request they are: the kubelet sorts by (usage − request) scaled by the request, so the Pod most egregiously over its promise dies first. A Burstable Pod within its request is treated as well-behaved and is much safer.
  3. Guaranteed Pods last — Pods with request == limit for every resource. They are evicted only as a final resort, when killing everything below them did not relieve the pressure.

Pod priority further modulates this within a QoS tier: among otherwise-equivalent candidates the kubelet prefers to evict the lower-priority Pod, so a high-priorityClassName Pod survives longer. The combined rule: QoS class is the primary key, usage-above-request is the secondary key, priority breaks ties.

Node-pressure eviction does NOT respect PodDisruptionBudgets

This is the fact that surprises people and deserves its own paragraph. A Pod Disruption Budget is a voluntary-disruption contract — it bounds how many Pods may be removed by planned operations (drains, autoscaler scale-down). Node-pressure eviction is not a planned operation; it is the node fighting for survival. The kubelet evicts whatever the ordering above selects, even if doing so drops an application below its PDB’s minAvailable. The only “protection” against node-pressure eviction is to be high in the eviction order — i.e. to be a Guaranteed Pod within its requests, ideally at high priority. PDBs do nothing here.

oom_score_adj and the kernel OOM kill — a third, related thing

Below kubelet eviction there is a fourth mechanism: the Linux kernel OOM killer. If memory is exhausted faster than the kubelet’s 10-second poll can react — a sudden allocation spike — the kernel acts first. To make the kernel’s choice align with Kubernetes’ QoS intent, the kubelet writes each container’s oom_score_adj (the /proc/<pid>/oom_score_adj knob the kernel uses to bias OOM victim selection):

  • BestEffort containers: oom_score_adj = 1000 — maximally killable; the kernel picks them first.
  • Burstable containers: a value between roughly 2 and 999, computed from how the container’s memory request compares to node capacity — more memory promised relative to the node, lower (safer) score.
  • Guaranteed containers: oom_score_adj = -997 — strongly protected; the kernel avoids them.

The distinction between kubelet node-pressure eviction and a kernel OOM kill:

Kubelet node-pressure evictionKernel OOM kill
Who actsThe kubelet (user-space)The Linux kernel
TimingProactive — on a 10 s poll, before memory is fully goneReactive — at the moment an allocation cannot be satisfied
GranularityEvicts a whole Pod, ranked by QoS + usageKills a single process in some cgroup, ranked by oom_score_adj
GraceSoft thresholds allow graceful termination; hard does notNone — SIGKILL, instantly
Pod statusstatus.reason: Evicted, Pod object retained for inspectionContainer lastState.terminated.reason: OOMKilled, exit 137
Respects PDBNoNo (not even an API-level concept here)

Well-tuned eviction thresholds aim to make the kubelet act before the kernel has to — eviction is at least somewhat orderly; a raw kernel OOM kill is not.

Configuration / API Surface

Node-pressure eviction is configured in the kubelet config (see kubelet for the full file):

# /var/lib/kubelet/config.yaml — eviction-relevant fields
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
 
evictionHard:                       # cross these ⇒ IMMEDIATE eviction, zero grace
  memory.available:  "100Mi"
  nodefs.available:  "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
  imagefs.inodesFree: "5%"
  pid.available:     "10%"
 
evictionSoft:                       # cross these for evictionSoftGracePeriod ⇒ graceful eviction
  memory.available:  "300Mi"
evictionSoftGracePeriod:
  memory.available:  "1m30s"        # signal must stay crossed this long before eviction
evictionMaxPodGracePeriod: 60       # cap on terminationGracePeriodSeconds during soft eviction
 
evictionMinimumReclaim:             # reclaim AT LEAST this much per eviction pass
  memory.available:  "200Mi"        #   prevents thrashing right at the threshold edge
  nodefs.available:  "1Gi"
 
mergeDefaultEvictionSettings: true  # keep built-in defaults when overriding a subset

API-initiated eviction is not configured on the kubelet — it is exercised via the API. The hand-rolled form:

# What `kubectl drain <node>` does for each Pod, under the hood:
curl -sS -X POST \
  https://<apiserver>/api/v1/namespaces/default/pods/web-7d9f-abc/eviction \
  -H 'Content-Type: application/json' \
  -d '{"apiVersion":"policy/v1","kind":"Eviction",
       "metadata":{"name":"web-7d9f-abc","namespace":"default"}}'
# → 200: eviction allowed, Pod gracefully deleted
# → 429: a PodDisruptionBudget would be violated — retry later

Diagnostics: kubectl get events -A --field-selector reason=Evicted surfaces node-pressure evictions; kubectl describe node <n> shows the MemoryPressure/DiskPressure/PIDPressure conditions; the kubelet metric kubelet_evictions counts evictions by signal.

Failure Modes

Eviction storm walking across the cluster. A node tips into memory pressure, the kubelet evicts a BestEffort Pod, the Pod’s controller immediately reschedules it onto another node, that node tips over, and the storm migrates. Symptom: a cascade of Evicted events across many nodes. Root cause is almost always unrealistic resource requests — requests set far below actual usage, so QoS collapses and the scheduler over-packs nodes. Fix: realistic requests/limits, a LimitRange supplying sane defaults, non-zero system-reserved/kube-reserved.

Guaranteed Pods getting evicted anyway. Operators assume Guaranteed QoS is eviction-proof. It is not — it is merely last in line. If BestEffort and over-request Burstable Pods together cannot relieve the pressure, the kubelet evicts Guaranteed Pods too. Fix: more headroom, fewer over-committed nodes; Guaranteed is a strong position, not an absolute shield.

kubectl drain hangs forever on 429. A PDB with minAvailable equal to (or above) the current healthy replica count means no Pod can be evicted without violating it — drain retries 429 indefinitely. Symptom: drain stuck on one Pod. Fix: scale the workload up first, relax the PDB, or (understanding the consequences) delete the Pod directly, bypassing the Eviction API.

Confusing OOMKilled with Evicted. OOMKilled (exit 137, container lastState) is a kernel action on one container that exceeded its memory limit — the Pod is not evicted, the container just restarts. Evicted (Pod status.reason) is a kubelet action that removes the whole Pod under node pressure. Mistaking one for the other sends debugging in the wrong direction. Fix: read the actual field — Pod-level reason: Evicted vs container-level reason: OOMKilled.

Disk-pressure eviction of the wrong Pod. A Pod writing huge emptyDir/log volumes drives nodefs pressure, but the kubelet’s reclaim (image GC, container GC) plus QoS-ordered eviction may sacrifice an unrelated BestEffort Pod first. Fix: ephemeral-storage requests/limits so the offender is itself an over-request target; quota log volumes.

Alternatives and When to Choose Them

The two eviction mechanisms are not interchangeable — each has its own role.

  • API-initiated eviction — use for planned Pod removal: node maintenance (kubectl drain), autoscaler scale-down, descheduling. It is the only mechanism that honours PDBs. Reach for it any time you are deliberately disrupting a workload.
  • Node-pressure eviction — not something you “choose”; it is the kubelet’s automatic last line of defence. You configure its thresholds, you do not invoke it.
  • Pod Priority and Preemption — a different mechanism for a different scarcity: the scheduler evicts (“preempts”) low-priority Pods to make room for a pending high-priority Pod that cannot otherwise be placed. Preemption is about admitting new Pods; node-pressure eviction is about saving a running node. Both consult priority, but they fire in different subsystems.
  • Plain kubectl delete pod — bypasses the Eviction API entirely, ignoring PDBs. Acceptable in emergencies when a drain is wedged on 429 and you accept the availability hit; never as routine tooling.

Production Notes

  • The eviction-storm pattern is documented across many post-mortems. Spotify and Shopify write-ups describe the same shape: one noisy under-requested workload starves a node, gets evicted, reschedules, and the storm walks the cluster. The fix is always the same — honest requests/limits, LimitRange defaults so nothing is accidentally BestEffort, and a non-zero system-reserved so the kubelet’s own working set is protected.
  • Set soft thresholds so the kubelet beats the kernel. A soft memory.available threshold with a short grace period lets the kubelet evict gracefully before the signal hits the hard threshold or the kernel OOM killer fires. Eviction is at least somewhat orderly; a raw OOM kill is a SIGKILL with no notice.
  • PDBs do not protect against node failure. A common mis-belief: “my PDB guarantees N replicas survive.” It guarantees that against drains and autoscaler scale-down only. Against node-pressure eviction, hardware failure, or a kernel OOM, the PDB is silent. Real availability needs replica count, anti-affinity / topology spread, and PDBs together.
  • evictionMinimumReclaim stops threshold thrashing. Without it, the kubelet evicts just enough to drop a hair below the threshold, the signal immediately re-crosses, and it evicts again. Setting a minimum reclaim makes each pass free up a meaningful chunk.
  • Watch kubelet_evictions and the per-signal pressure metrics. Eviction is a symptom; the metrics tell you which resource and how often, which points at whether the fix is more memory, more disk, or better requests.

See Also

Graph View

Backlinks