vault

Pod Priority and Preemption

10 min read

Pod Priority and Preemption

Pod priority gives Pods a relative importance ranking; preemption is what the scheduler does with that ranking when the cluster is full. A PriorityClass is a cluster-scoped object that maps a name to an integer priority value; a Pod opts in via spec.priorityClassName, and an admission controller stamps the resolved integer onto spec.priority. Priority does two things: it orders the scheduler’s queue (higher-priority pending Pods are attempted first), and — when a high-priority Pod cannot fit anywhere — it triggers preemption: the scheduler evicts lower-priority Pods from a node to make room. Preemption is implemented as the DefaultPreemption PostFilter plugin of the kube-scheduler — it runs only after the Filter phase has found zero feasible nodes. The feature reached GA in Kubernetes v1.14 (Pod Priority and Preemption); the preemptionPolicy: Never (non-preempting) refinement reached GA in v1.24.

Mental Model

Priority converts the cluster from “first-Pod-to-arrive wins the last slot” into “most-important-Pod wins the last slot — even if it has to throw someone out.”

flowchart TB
    P["High-priority Pod P (priority 1,000,000)<br/>requests 4 CPU"] --> SCHED["Scheduling cycle"]
    SCHED --> F{"Filter: any feasible node?"}
    F -->|"yes"| BIND["Bind normally"]
    F -->|"no — cluster full"| PF["PostFilter: DefaultPreemption"]
    PF --> SCAN["Scan nodes: where would evicting<br/>lower-priority Pods free 4 CPU?"]
    SCAN --> PICK["Pick node N; victims = lowest-priority<br/>Pods that together free enough"]
    PICK --> PDB{"Would eviction<br/>violate a PDB?"}
    PDB -->|"avoidable"| PREFER["Choose victims that respect PDBs"]
    PDB -->|"unavoidable"| ANYWAY["Preempt anyway — best-effort only"]
    PREFER --> EVICT["Gracefully delete victims<br/>(terminationGracePeriodSeconds)"]
    ANYWAY --> EVICT
    EVICT --> NOM["Set P.status.nominatedNodeName = N"]
    NOM --> REQ["Requeue P for next cycle"]

What this diagram shows. Preemption is not part of the normal happy path — it fires only in the PostFilter phase, after Filter has exhausted every node. The scheduler then searches for a node where evicting the lowest-priority Pods would free enough room, prefers victim sets that keep Pod Disruption Budgets intact, but will violate a PDB if no PDB-respecting option exists (preemption respects PDBs only on a best-effort basis). Victims are deleted gracefully, honoring their terminationGracePeriodSeconds. The insight to extract: preemption does not place P immediately — it merely clears space and sets nominatedNodeName as a hint; P is then requeued and goes through a fresh scheduling cycle, and is not guaranteed to land on the nominated node (another Pod, or another scheduler, may take it in the interval).

Mechanical Walk-through

The PriorityClass object

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "Latency-critical user-facing services."

value is a signed 32-bit integer; user-defined classes are conventionally capped at 1,000,000,000 (the range above that is reserved). name must be a DNS subdomain and must not start with system-. Setting globalDefault: true makes this class the priority of any Pod that does not name a class — only one PriorityClass in the cluster may be globalDefault. If no globalDefault exists, Pods without a priorityClassName get priority 0.

The two reserved system classes

Kubernetes ships two PriorityClasses you cannot delete and should not imitate:

  • system-node-critical (value 2000001000) — the highest priority in the cluster, for Pods that must run on every node or the node is useless: the CNI agent, kube-proxy, the node-problem-detector.
  • system-cluster-critical (value 2000000000) — for control-plane-adjacent Pods that must run somewhere: CoreDNS, the metrics-server.

These values sit above the 1,000,000,000 ceiling for user classes precisely so no application Pod can ever out-prioritize cluster infrastructure and preempt it.

How priority orders the queue

The scheduler’s QueueSort plugin (PrioritySort by default) keys the active queue on spec.priority descending, with creation timestamp as the tiebreaker. A burst of high-priority Pods therefore jumps ahead of a backlog of low-priority ones — before any preemption is even considered. Priority affects ordering even when the cluster has room; preemption is the additional behavior when it does not.

The preemption algorithm

When Filter yields no feasible node, DefaultPreemption runs:

  1. For each node, compute a hypothetical: which lower-priority Pods would have to be removed for the pending Pod P to fit? A Pod is a candidate victim only if its priority is strictly lower than P’s.
  2. Among nodes where preemption could work, prefer the one that: minimizes the number of PDB violations, then minimizes the highest victim priority, then minimizes the sum of victim priorities, then evicts the fewest Pods.
  3. Mark the chosen victims for graceful deletion. They receive their full terminationGracePeriodSeconds, so a victim with a 30 s grace period and a preStop hook gets to drain.
  4. Set P.status.nominatedNodeName to the chosen node. This is a hint, visible in kubectl describe pod, that also tells other pending Pods’ scheduling cycles that this node’s freed capacity is spoken for.
  5. Requeue P. P is not bound during the PostFilter — it re-enters the queue and is scheduled afresh once the victims have actually terminated.

preemptionPolicy — preempting vs non-preempting

Set on the PriorityClass:

  • PreemptLowerPriority (default) — Pods of this class do preempt lower-priority Pods when they cannot fit.
  • Never — Pods of this class are placed ahead in the queue (they still enjoy the priority ordering benefit) but never evict anyone. They wait for capacity to free up naturally. They can themselves still be preempted by a higher-priority PreemptLowerPriority Pod. This is the right setting for important-but-not-urgent batch/data-science work: jump the queue, but never discard a running job to do it.

Preemption and PodDisruptionBudget

The scheduler’s victim-selection step attempts to choose Pods such that no Pod Disruption Budget is violated. But the guarantee is best-effort: if the only way to fit a high-priority Pod is to evict a PDB-protected Pod, the scheduler will violate the PDB. The Kubernetes docs are explicit that preemption “tries its best to respect PDBs, but … if no PDB-violation-free node is available, preemption may still happen by violating PDBs.” A PDB protects against voluntary disruption (drains); preemption sits in a gray zone and the high-priority Pod wins.

Configuration / API Surface

A non-preempting batch class and a Pod that uses it:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-elevated
value: 100000                       # well below user-facing services
globalDefault: false
preemptionPolicy: Never             # jump the queue, but DON'T evict anyone
description: "Nightly ETL — important, but must not kill live traffic."
---
apiVersion: v1
kind: Pod
metadata:
  name: nightly-etl
spec:
  priorityClassName: batch-elevated # resolved to value 100000 by the
                                    # Priority admission controller
  containers:
  - name: etl
    image: etl:v9
    resources:
      requests:
        cpu: "8"
        memory: "16Gi"
  terminationGracePeriodSeconds: 120 # if a HIGHER-priority Pod preempts THIS
                                     # one, it gets 120s to checkpoint

Line-by-line. value: 100000 ranks this work above unclassified Pods (priority 0) but far below latency-critical services — so it will not delay them in the queue. preemptionPolicy: Never is the load-bearing field: this ETL job will be attempted before priority-0 Pods, but if the cluster is full it waits rather than killing live workloads. The Pod references the class by name only; the Priority admission controller does the name-to-integer resolution at admission time (a Pod naming a non-existent class is rejected). terminationGracePeriodSeconds: 120 matters for the converse case: if a system-cluster-critical or higher-priority Pod ever preempts this ETL Pod, the 120 s window lets it checkpoint progress before SIGKILL.

Failure Modes

Preemption thrash. A high-priority Pod cannot fit → it preempts a lower-priority Pod → that Pod’s controller recreates it → the recreated Pod cannot fit and triggers another preemption elsewhere → repeat. The cluster churns without converging. Mitigations: keep the number of distinct priority levels small (a handful, not dozens); ensure real headroom; consider preemptionPolicy: Never for classes that should wait rather than fight.

Priority-inversion / unintended global default. Someone creates a PriorityClass with globalDefault: true and a high value; suddenly every unclassified Pod cluster-wide is high priority and preemption logic engages everywhere. globalDefault should be used rarely and conservatively (often not at all, or a value of 0).

PDB silently violated. An operator assumes a PDB of minAvailable: 2 guarantees two replicas survive; a high-priority Pod preempts down to one because no PDB-respecting victim set existed. The PDB protected against drains, not preemption. Diagnostic: correlate Preempted events with the protected Deployment.

Victims take too long to die. Preemption marks victims for graceful deletion; if a victim has a long terminationGracePeriodSeconds or a stuck preStop hook, the high-priority Pod stays Pending for that whole window. The nominated node is “reserved” but not yet free.

Malicious priority escalation. In a multi-tenant cluster, a tenant who can create Pods with a high priorityClassName can preempt other tenants’ workloads at will. Mitigation: gate PriorityClass usage with a ResourceQuota scoped by scopeSelector: priorityClass, or an admission policy (Kyverno / OPA Gatekeeper).

Nominated node not used. nominatedNodeName is only a hint; between PostFilter and the requeue, another Pod may take the freed capacity. The high-priority Pod then re-preempts elsewhere. This is normal but inflates scheduling latency.

Alternatives and When to Choose Them

  • Taints and Tolerations / Node Affinity — they decide where a Pod may run; priority decides whose Pod wins when space is contested. Different axes — a high-priority Pod still obeys taints and affinity.
  • Cluster Autoscaler / Karpenter — the “add capacity” answer to “the cluster is full,” versus preemption’s “take capacity from someone.” In practice they cooperate: preemption gives the urgent Pod a slot now, while the autoscaler adds a node for the evicted Pod. Karpenter is also priority-aware in its provisioning decisions.
  • Gang schedulers (Volcano, Kueue) — for batch/HPC where all-or-nothing placement of a Pod group matters, plain Pod priority is insufficient (it can preempt half a job). Volcano adds priority-aware gang preemption. See Batch Workloads on Kubernetes.
  • preemptionPolicy: Never — the in-feature alternative to full preemption: get queue priority without the eviction side effects.

Production Notes

  • Keep priority levels few and meaningful. The recurring industry advice (echoed in scheduler-scaling write-ups from Spotify and OpenAI) is 3–5 named classes — e.g., system-critical (reserved), user-facing, internal-services, batch, best-effort. Dozens of fine-grained values make preemption behavior impossible to predict.
  • Protect cluster infrastructure with the reserved classes. CNI, kube-proxy, CoreDNS, and monitoring DaemonSets should carry system-node-critical / system-cluster-critical so no application Pod can preempt them — a real cause of cascading outages when forgotten.
  • Pair priority with ResourceQuota-by-priority-class in any multi-tenant cluster, or a tenant can starve others by flooding high-priority Pods.
  • Preemption is not eviction. Distinguish it from Pod Eviction (node-pressure eviction by the kubelet, driven by QoS Classes) and from API-initiated eviction (drains, Pod Disruption Budget). Three different mechanisms remove Pods for three different reasons; conflating them produces bad incident postmortems.
  • Monitor scheduler_preemption_attempts_total and scheduler_preemption_victims. A rising victim count is the early signal of thrash or chronic under-capacity.

See Also

Graph View

Backlinks