Taints and Tolerations

Taints and tolerations are the repulsion half of Kubernetes scheduling — the inverse of Node Affinity. A taint is a node-side marker (key=value:Effect) that says “keep Pods off me”; a toleration is a Pod-side token that says “I consent to run on a node carrying this taint.” The two must match for a Pod to be admissible. Affinity attracts a Pod toward nodes; taints repel every Pod that has not explicitly opted in. The crucial asymmetry: a toleration does not guarantee a Pod lands on a tainted node — it merely makes the node eligible. To force a Pod onto specially-marked nodes you combine a taint (to keep everyone else off) with a toleration and Node Affinity (to pull the chosen Pods on). Taints are evaluated by the scheduler’s TaintToleration Filter plugin (Taints and Tolerations, kube-scheduler).

Mental Model

A taint is best thought of as a node saying “no” by default and the toleration as one specific Pod’s signed waiver for that exact “no.” A node may carry many taints; a Pod’s set of tolerations is checked against all of them, and a single un-tolerated taint with a blocking effect is enough to reject (or evict) the Pod.

flowchart LR
    subgraph NODE["Node: gpu-node-1"]
        T1["taint: nvidia.com/gpu=true:NoSchedule"]
        T2["taint: node.kubernetes.io/disk-pressure:NoSchedule<br/>(auto-applied by kubelet)"]
    end
    PODA["Pod A<br/>tolerations: [nvidia.com/gpu]"]
    PODB["Pod B<br/>tolerations: []"]
    PODC["Pod C<br/>tolerations: [nvidia.com/gpu, disk-pressure]"]
    PODA -. "rejected: disk-pressure<br/>taint not tolerated" .-> NODE
    PODB -. "rejected: neither<br/>taint tolerated" .-> NODE
    PODC == "admitted: every taint<br/>has a matching toleration" ==> NODE

What this diagram shows. A node with two taints — one set deliberately by an operator (nvidia.com/gpu), one auto-applied by the kubelet because the node is under disk pressure. The matching rule is all-or-nothing: Pod A tolerates the GPU taint but not the disk-pressure taint, so it is still rejected; Pod B tolerates nothing; only Pod C, which tolerates every taint on the node, is admitted. The insight to extract: tolerations are evaluated conjunctively against the node’s full taint set — adding a toleration removes exactly one obstacle, not all of them, and auto-applied condition taints can silently block a Pod the operator believed was correctly configured.

Mechanical Walk-through

Applying a taint

A taint is added to a node with kubectl taint:

kubectl taint nodes gpu-node-1 nvidia.com/gpu=true:NoSchedule

The grammar is key=value:Effect. The value is optional (key:Effect is legal — it pairs with a toleration using operator: Exists). Removing a taint appends a trailing hyphen:

kubectl taint nodes gpu-node-1 nvidia.com/gpu=true:NoSchedule-

Taints live in node.spec.taints[]; kubectl describe node prints them under Taints:.

The three effects

The Effect field controls what the taint does:

• NoSchedule — a hard scheduling filter. The scheduler will not place a new Pod on the node unless the Pod tolerates the taint. Already-running Pods are not touched. This is the workhorse effect for dedicated-node patterns.
• PreferNoSchedule — a soft version. The scheduler tries to avoid placing non-tolerating Pods here but will do so if no untainted node is available. It contributes a score penalty rather than a Filter rejection — the analogue of preferredDuringScheduling for Node Affinity.
• NoExecute — the aggressive effect. It does everything NoSchedule does and evicts already-running Pods that do not tolerate it. A Pod that does tolerate a NoExecute taint stays bound forever — unless it sets tolerationSeconds (see below).

tolerationSeconds — the eviction timer

For a NoExecute toleration only, the Pod may set tolerationSeconds: N. This means “I tolerate this taint, but only for N seconds after it appears; then evict me.” If the taint is removed before the timer expires, the Pod is not evicted. This single field is the mechanism behind the platform’s default “wait 5 minutes before evicting Pods from a NotReady node” behavior: the node-lifecycle controller of kube-controller-manager applies node.kubernetes.io/not-ready:NoExecute, and the admission controller stamps every Pod with a default tolerationSeconds: 300 toleration for it. A NotReady blip shorter than 300 s therefore causes no eviction; a sustained outage evicts the Pods so a controller can reschedule them elsewhere.

Built-in taints — the ones the platform applies for you

The kubelet and node-lifecycle controller automatically apply condition taints that mirror node conditions (Node conditions):

Taint key	Applied when	Typical effect
node.kubernetes.io/not-ready	Node Ready condition is False	NoExecute (+ NoSchedule)
node.kubernetes.io/unreachable	Node controller lost contact (Ready = Unknown)	NoExecute (+ NoSchedule)
node.kubernetes.io/memory-pressure	Kubelet reports MemoryPressure	NoSchedule
node.kubernetes.io/disk-pressure	Kubelet reports DiskPressure	NoSchedule
node.kubernetes.io/pid-pressure	Kubelet reports PIDPressure	NoSchedule
node.kubernetes.io/unschedulable	Node cordoned (kubectl cordon)	NoSchedule
node.kubernetes.io/network-unavailable	CNI has not finished wiring the node	NoSchedule

The controllers then add matching tolerations to system-critical Pods (and DaemonSet Pods get broad tolerations automatically) so cluster infrastructure is not evicted by its own node-pressure taints. This is why a DaemonSet log shipper keeps running on a disk-pressured node while ordinary workloads are kept off it.

Matching rules — operators and the conjunctive check

A toleration matches a taint when the key and effect agree and one of:

• operator: Equal (the default) — the value fields also agree.
• operator: Exists — no value is compared; the toleration matches the key regardless of value. An Exists toleration with an empty key and empty effect tolerates every taint — the “tolerate everything” pattern used by some monitoring DaemonSets.

The scheduler starts with the node’s full taint list, discards every taint that has a matching toleration, and then inspects what remains: any leftover NoSchedule or NoExecute taint rejects the Pod; any leftover PreferNoSchedule adds a soft penalty.

Configuration / API Surface

A Pod scheduled onto a dedicated GPU node — taint + toleration + nodeAffinity together:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-trainer
spec:
  # ---- consent to the node's taint ----
  tolerations:
  - key: "nvidia.com/gpu"          # must match the taint key
    operator: "Equal"              # compare value too (default)
    value: "true"                  # must match the taint value
    effect: "NoSchedule"           # must match the taint effect
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 60          # evict 60s after node goes unreachable,
                                   # not the default 300s — fail fast
  # ---- pull the Pod ONTO the GPU nodes (taint alone won't do this) ----
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: "nvidia.com/gpu"
            operator: Exists       # only nodes labelled as GPU nodes
  containers:
  - name: trainer
    image: training-job:v3
    resources:
      limits:
        nvidia.com/gpu: 1          # extended resource; see [[Resource Requests and Limits]]

Line-by-line. The first toleration waives the operator-set nvidia.com/gpu=true:NoSchedule taint — without it, the scheduler’s TaintToleration Filter rejects every GPU node. The second toleration overrides the default 300 s unreachable toleration with a tighter 60 s window: this Pod should be rescheduled fast if its node drops off the network, accepting a higher false-positive eviction rate. The nodeAffinity block is the load-bearing complement: a toleration only makes GPU nodes eligible; without affinity the scheduler could equally place this Pod on a cheap CPU node. The pairing — taint repels everyone else, toleration grants this Pod a waiver, affinity attracts this Pod specifically — is the canonical dedicated-node pattern.

Failure Modes

Pod stuck Pending despite free capacity. kubectl describe pod shows FailedScheduling … node(s) had untolerated taint {key: value}. The Pod is missing a toleration — often for an auto-applied taint the operator forgot about (memory-pressure on a strained node, an unschedulable taint from a forgotten kubectl cordon).

Workload disappears from a node after an incident. A NoExecute taint was applied (manually for maintenance, or automatically because the node went NotReady) and the Pods did not tolerate it. Diagnostic: kubectl get events --field-selector reason=TaintManagerEviction.

Dedicated node “leaks” other workloads. The node was tainted but a broadly-permissioned Pod (a DaemonSet with a catch-all Exists toleration, or a Pod copied from a template that carried a stale toleration) tolerates the taint and lands there anyway. The taint keeps the majority off but is not a hard partition — for true isolation pair it with a ResourceQuota and admission policy.

PreferNoSchedule ignored. Operators sometimes expect PreferNoSchedule to keep nodes empty; it only nudges. Under capacity pressure non-tolerating Pods will land there. If you need a guarantee, use NoSchedule.

Eviction storm after a control-plane hiccup. If the node-lifecycle controller marks many nodes unreachable at once (e.g., an apiserver overload), every non-tolerating Pod on those nodes is NoExecute-evicted simultaneously. The --large-cluster-size-threshold and the controller’s eviction-rate limiter exist to dampen this.

Alternatives and When to Choose Them

• Node Affinity / Node Selector — the attraction mechanism. Affinity expresses “this Pod wants node X”; taints express “node X rejects Pods by default.” They are complementary, not substitutes: affinity alone does not stop other Pods from also landing on the node, and taints alone do not pull the intended Pod on. Dedicated nodes need both.
• Pod Affinity and Anti-Affinity — repulsion/attraction relative to other Pods, not to node labels. Use anti-affinity to keep replicas apart; use taints to keep workloads off reserved hardware.
• Topology Spread Constraints — even distribution across domains. Orthogonal to taints: a Pod can be spread across zones and be kept off tainted nodes.
• ResourceQuota + admission policy — for enforced multi-tenant isolation. Taints are advisory in the sense that any Pod author can add a toleration; a quota or Kyverno/OPA Gatekeeper policy is what actually prevents a tenant from tolerating another tenant’s nodes.

Production Notes

• Managed-Kubernetes node pools taint by default. GKE, EKS, and AKS let a node pool declare taints at creation; GPU and Spot/Preemptible pools are almost always tainted so only opted-in workloads land there. EKS Managed Node Groups and GKE node-pool --node-taints are the common entry points.
• The classic outage: a kubectl cordon (which stamps node.kubernetes.io/unschedulable:NoSchedule) is left in place after maintenance; weeks later a scale-up event finds “mysteriously” unschedulable capacity. Always pair cordon with a follow-up uncordon, or script it.
• Keep the PriorityClass and taint counts small. Dozens of bespoke taints across a fleet make it impossible to reason about where a Pod can run; a handful of well-named taint keys (dedicated, gpu, spot) is the maintainable shape.
• DaemonSets and tolerations: the DaemonSet controller automatically adds tolerations for the node-condition taints so node agents survive node pressure — but it does not add tolerations for your custom taints. A monitoring DaemonSet that must run on a tainted GPU pool needs the toleration spelled out explicitly.
• tolerationSeconds tuning is a latency-vs-stability dial: lower it for stateless workloads that should fail over fast; keep the default 300 s (or raise it) for workloads where a brief node blip is cheaper than a reschedule.

See Also

• Node Affinity — the attraction counterpart; dedicated-node pattern needs both
• Node Selector — the simpler attraction primitive
• Pod Affinity and Anti-Affinity — repulsion/attraction relative to other Pods
• Topology Spread Constraints — even spread across domains
• kube-scheduler — runs the TaintToleration Filter plugin
• Pod Priority and Preemption — the other “evict to make room” mechanism
• Pod Eviction — node-pressure eviction, distinct from NoExecute taint eviction
• Node Pressure Conditions — the conditions that trigger auto-applied taints
• DaemonSet — gets node-condition tolerations automatically
• Resource Requests and Limits — extended resources like nvidia.com/gpu
• Kubernetes Node — where spec.taints[] lives
• Kubernetes MOC — §9 Scheduling