vault

kube-scheduler

14 min read

kube-scheduler

The kube-scheduler is the control-plane component responsible for assigning newly created Pods (those with no spec.nodeName set) to specific Nodes, by running each Pod through a two-phase pipeline of Filter (predicates) then Score (priorities) and writing the selected node back to the apiserver via the binding subresource (Kubernetes — Components, kube-scheduler concepts). Since Kubernetes 1.19 the scheduler has been architected around the Scheduling Framework (KEP-624), a pluggable system of ten extension points (PreFilter, Filter, PostFilter, PreScore, Score, NormalizeScore, Reserve, Permit, PreBind, Bind, PostBind, plus PreEnqueue and QueueSort) that every scheduling decision passes through. The scheduler does not start containers or even reach out to nodes — it is purely a decision engine that converts unscheduled-Pod events into Pod-to-Node bindings, with kubelet then reading the assignment and instructing the runtime to actually run the workload. Operationally the scheduler is a perpetually backlog-driven loop: when its unschedulable backlog rises, the Cluster Autoscaler watches the same signal and adds nodes; when the backlog drops, the cluster is in steady state.

Mental Model

The scheduler is a queue-driven optimization loop: pull the next Pod from a priority queue, filter the cluster’s nodes down to those that can host it, score the survivors, and bind to the highest-scoring node. Crucially it is single-threaded over scheduling decisions per profile — the scheduling cycle runs serially so that one decision cannot trample another’s reservations — but the per-cycle work (filtering many nodes, scoring many nodes) is parallelized, and the post-decision binding cycle runs asynchronously so the scheduler can move to the next Pod while etcd’s write completes (Scheduling Framework).

flowchart LR
    UNSCHED[Unscheduled Pods<br/>watched via apiserver] --> ACTIVE[Active Queue<br/>QueueSort by priority + age]
    BACKOFF[Backoff Queue<br/>exponential retry delay]
    UNQUEUE[Unschedulable Queue<br/>recently failed Pods]
    BACKOFF -->|backoff elapsed| ACTIVE
    UNQUEUE -->|QueueingHint:<br/>cluster event matches| ACTIVE
    ACTIVE --> CYCLE[Scheduling Cycle<br/>serial]
    CYCLE --> PF[PreFilter]
    PF --> F[Filter<br/>per-node, parallel]
    F -->|no feasible node| POSTF[PostFilter<br/>typically Preemption]
    F --> PS[PreScore]
    PS --> S[Score<br/>per-node, parallel]
    S --> NS[NormalizeScore]
    NS --> SEL[Select highest-scoring node]
    SEL --> R[Reserve]
    R --> P[Permit<br/>can Wait, Approve, Deny]
    P --> BIND[Binding Cycle<br/>async]
    BIND --> PB[PreBind]
    PB --> B[Bind<br/>POST /pods/X/binding]
    B --> POSTB[PostBind]
    POSTF -.->|requeue| UNQUEUE
    P -.->|Deny| UNQUEUE
    B -.->|error| UNQUEUE

What this diagram shows. Top-left: the three queues that hold Pods between scheduling attempts. The active queue is the work-stealing priority queue ordered by Pod priority and arrival; the backoff queue holds Pods that failed recently with exponential delay; the unschedulable queue holds Pods that failed for a structural reason (no node large enough) and waits for a cluster event (a new node, a Pod deleted) that might make them schedulable — the QueueingHint mechanism, stable in 1.34, decides per-event whether a Pod’s situation actually changed. Top-right: the scheduling cycle is one serial pipeline per Pod; the binding cycle runs asynchronously so multiple bindings can be in flight at once. The insight to extract is that the scheduler does not “place every Pod optimally”; it places each Pod locally optimally given the cluster state at the moment of that decision, and the post-fact correctness depends on bookkeeping (Reserve / Permit) plus retry logic when reality contradicts the decision (e.g., another scheduler beat it to a binding).

Mechanical Walk-through

A Pod enters the scheduler’s universe by being created in etcd with no spec.nodeName. The scheduler maintains an informer on Pods filtered by spec.nodeName="" and on every node-affecting resource (Nodes, PersistentVolumes, StorageClasses, ResourceClaims). When the Pod-watch fires, the new Pod runs through PreEnqueue plugins; if any returns non-Success, the Pod goes to the unschedulable queue without an Unschedulable condition (the canonical use case is “wait for a CRD to be ready before considering this Pod”). Otherwise it enters the active queue, ordered by the QueueSort plugin (the default uses Pod priority as the primary key and creation timestamp as tie-breaker).

The scheduler pops the head of the active queue and starts a scheduling cycle. The cycle is purely in-memory and serial — exactly one Pod is in the cycle at any moment per scheduler profile. PreFilter runs first, computing shared state used by Filter (e.g., the InterPodAffinity plugin walks every Pod once to build its affinity term index). If PreFilter returns Unschedulable, the cycle aborts and the Pod goes to the unschedulable queue.

Filter runs next, per-node in parallel. Each Filter plugin tests whether the node satisfies a hard constraint: NodeResourcesFit checks that requested CPU and memory plus the Pod’s overhead fit in the node’s allocatable resources minus already-assigned Pods; NodeAffinity evaluates spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution; TaintToleration rejects tainted nodes that the Pod does not tolerate; VolumeBinding rejects nodes that cannot host the Pod’s PVC zone; PodTopologySpread rejects nodes that would violate spread constraints. The scheduler stops early on a node as soon as one Filter rejects it.

To bound the per-cycle Filter cost, the scheduler uses the percentageOfNodesToScore mechanism: instead of evaluating every node, it walks the node list in a round-robin order (interleaved across zones for fairness) and stops once it has found enough feasible nodes to make a reasonable scoring decision. The default is automatic and linear: ~50% at 100 nodes scaling to 10% at 5000 nodes, with a 5% floor (Scheduler Performance Tuning). This is the single most important performance lever for large clusters and the dial behind “why isn’t the scheduler placing my Pods on the least-loaded node” surprises — it might not have looked at the least-loaded node at all.

If Filter produced zero feasible nodes, PostFilter runs. The canonical PostFilter plugin is DefaultPreemption: it walks nodes looking for one where evicting lower-priority Pods would let the high-priority Pod fit, marks the victims for graceful eviction, and rerques the Pod for the next cycle. Pod priority and preemption together form a global ordering: a high-priority Pod that can’t fit anywhere will displace lower-priority Pods rather than waiting for a new node.

If Filter found feasible nodes, PreScore runs (per-Pod once), then Score runs per-node in parallel. Each Score plugin returns an int in [MinNodeScore, MaxNodeScore] (0 to 100 by default). The Score plugins enabled by default are, per the scheduler config reference (Scheduler Configuration): ImageLocality, TaintToleration, NodeAffinity, PodTopologySpread, NodeResourcesFit, NodeResourcesBalancedAllocation, VolumeBinding, and InterPodAffinity. Reading the important ones: NodeResourcesFit rewards a chosen resource profile (multiple strategies — LeastAllocated, MostAllocated, RequestedToCapacityRatio); NodeResourcesBalancedAllocation rewards nodes whose CPU and memory utilization end up balanced rather than lopsided; InterPodAffinity rewards co-location with Pods matching the affinity terms; PodTopologySpread rewards spreading across topology domains; ImageLocality rewards nodes that already have the Pod’s image cached; NodeAffinity rewards preferredDuringSchedulingIgnoredDuringExecution matches; TaintToleration rewards nodes whose taints the Pod tolerates with PreferNoSchedule. NormalizeScore rescales each plugin’s scores to a common range so weighting can combine them. The scheduler then computes a weighted sum per node, picks the maximum, breaks ties randomly, and proceeds to Reserve.

Reserve notifies all stateful plugins that resources on the selected node are reserved for this Pod — for example, the VolumeBinding plugin reserves the PVC-to-Node binding so a parallel scheduling attempt doesn’t double-book the same volume. Permit is the final checkpoint where a plugin can return Approve, Deny, or Wait (the last allows a callback-driven delay — used by gang-scheduling plugins like Volcano that wait for all members of a Job to be schedulable before binding any).

Once Permit approves, the scheduling cycle ends and a binding cycle begins asynchronously. PreBind does any pre-binding work (PVC provisioning waits for dynamic provisioning to complete here). Bind writes the Binding subresource — equivalent to POST /api/v1/namespaces/<ns>/pods/<name>/binding with the chosen node — and the apiserver patches spec.nodeName on the Pod. PostBind runs informational hooks. If any binding-cycle stage fails, Unreserve runs every plugin in reverse order (idempotently) to release reservations, and the Pod is requeued.

In a HA control plane, multiple scheduler replicas run with leader election via a Lease object in kube-system (default name kube-scheduler); only the leader actively schedules, the others stand by (Leases).

Configuration / API Surface

A representative KubeSchedulerConfiguration for a 1000-node cluster with a custom plugin enabled:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
  resourceLock: leases
  resourceName: kube-scheduler
  resourceNamespace: kube-system
percentageOfNodesToScore: 30
parallelism: 32
profiles:
  - schedulerName: default-scheduler
    plugins:
      score:
        enabled:
          - name: NodeResourcesFit
            weight: 1
          - name: PodTopologySpread
            weight: 2
          - name: InterPodAffinity
            weight: 1
          - name: ImageLocality
            weight: 1
        disabled:
          - name: NodeResourcesBalancedAllocation
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: LeastAllocated
            resources:
              - name: cpu
                weight: 1
              - name: memory
                weight: 1
  - schedulerName: batch-scheduler
    plugins:
      queueSort:
        enabled:
          - name: PrioritySort
      score:
        disabled:
          - name: "*"
        enabled:
          - name: NodeResourcesFit
            weight: 1
      pluginConfig:
        - name: NodeResourcesFit
          args:
            scoringStrategy:
              type: MostAllocated  # pack tightly for batch

Line-by-line. leaderElection.leaderElect: true enables the multi-replica leader-election dance; resourceLock: leases uses the modern coordination.k8s.io/v1 Lease object (deprecated alternatives were endpoints and configmaps). percentageOfNodesToScore: 30 overrides the default linear formula. That formula yields 50% at 100 nodes and 10% at 5000 nodes with a 5% floor (Scheduler Performance Tuning); linearly interpolated, at 1000 nodes the automatic value is roughly 43%, so pinning it to 30% trades a slightly narrower search for lower, more predictable per-cycle latency. parallelism: 32 is the size of the goroutine pool used for parallel Filter and Score (default 16); higher values use more CPU per cycle but reduce latency. The two profiles show how to run multiple schedulers in one process — Pods set spec.schedulerName: batch-scheduler to opt into the MostAllocated packing strategy for batch jobs, while regular workloads get the default LeastAllocated spread. The Score section enables a custom weighted combination — note PodTopologySpread is weighted 2× to emphasize zone-spreading. The pluginConfig blocks pass plugin-specific arguments; NodeResourcesFit.args.scoringStrategy controls the bin-packing heuristic.

To use a fully custom scheduler (e.g., Volcano for gang scheduling, Karpenter’s scheduling logic, or a research scheduler), deploy it as a separate Deployment with spec.template.spec.schedulerName: my-scheduler and have Pods opt in by setting spec.schedulerName: my-scheduler — the default scheduler ignores Pods scheduled by another scheduler (Configure Multiple Schedulers).

Failure Modes

Unschedulable backlog. The scheduler’s scheduler_pending_pods{queue="unschedulable"} metric climbs and stays high. Causes are (1) cluster genuinely full → Cluster Autoscaler should add nodes, (2) misconfigured Pod requests that no node can satisfy (request 32 vCPU when largest node is 16), (3) PVC zone mismatch where the Pod needs zone A but no nodes in zone A have capacity. The diagnostic is kubectl get events --field-selector reason=FailedScheduling plus the Pod’s status.conditions[type=PodScheduled].message which spells out which Filter plugins rejected it (“0/120 nodes are available: 80 Insufficient cpu, 40 node(s) had untolerated taint”).

Pending Pods despite available capacity. Symptom: nodes have free CPU and memory but Pods stay Pending. Causes are usually subtle: PodTopologySpread constraints that would violate at the current node, anti-affinity terms whose label set unexpectedly matches running Pods, or PVCs bound to a zone with no available nodes. The percentageOfNodesToScore setting being too low can also cause this — the scheduler may not have evaluated the right nodes.

Preemption thrash. A high-priority Pod cannot fit anywhere → preemption evicts a lower-priority Pod → the evicted Pod is recreated by its controller → it now cannot find a node and triggers another preemption. Mitigations: tighten PriorityClass count (a few priorities, not dozens), set preemptionPolicy: Never on Pods that shouldn’t trigger preemption, ensure cluster has headroom.

Slow scheduling latency at scale. scheduler_e2e_scheduling_duration_seconds p99 climbs past 1 second. The dominant cost is usually Filter on NodeResourcesFit and PodTopologySpread; the fix is percentageOfNodesToScore reduction or enabling opportunistic batching (1.35+ beta) which caches Filter/Score results for identical Pods (Scheduler Performance Tuning).

Leader election thrash. Two scheduler replicas alternate holding the lease. Cause is usually an overloaded apiserver (the leader’s lease renewal request times out) or network partition. Each leader change costs a fresh informer sync. Diagnose via leader_election_master_status metric.

Binding races. Two parallel binding cycles end up writing the same node-port allocation or PVC binding. The apiserver’s optimistic-concurrency 409 returned to one cycle triggers Unreserve and a requeue. This is normal; if the failure rate exceeds a few percent, investigate cluster size vs. parallelism.

Alternatives and When to Choose Them

The kube-scheduler is the default; alternatives target specific workload classes:

  • Volcano (CNCF Incubating): the canonical batch / HPC / ML scheduler. Adds gang scheduling (all-or-nothing for a group of Pods), queue-based fair scheduling, and co-scheduling for distributed training jobs. Used by every major Kubeflow / Ray-on-K8s deployment for multi-Pod jobs. Choose when running tightly-coupled distributed workloads where partial scheduling is worthless.
  • Yunikorn (Apache, originally LinkedIn): hierarchical queue scheduler with rich resource sharing semantics, modeled on YARN’s CapacityScheduler. Choose for multi-tenant clusters where teams need guaranteed slices of capacity.
  • Karpenter (AWS-born, multi-cloud): not strictly a scheduler — it is a node provisioner that watches the default scheduler’s unschedulable backlog and provisions exactly the instance type that would let the pending Pods fit, then lets the default scheduler bind. Replaces the Cluster Autoscaler with a faster, bin-packing-aware alternative. See Karpenter.
  • Custom schedulers via the framework: write a Go plugin against k8s.io/kubernetes/pkg/scheduler/framework, register it via KubeSchedulerConfiguration, and either replace or augment the default profile. Examples: cost-aware scheduling, hardware-topology-aware GPU scheduling, latency-aware service-mesh-aware scheduling.
  • Out-of-process custom schedulers (run alongside the default): write a controller that watches spec.schedulerName: my-scheduler Pods and writes Bindings via the API. Simpler to deploy than a framework plugin but loses access to the scheduler’s optimized informer cache.

Production Notes

The scheduler’s behavior at scale is documented in the large-cluster best practices and discussed in various engineering blogs:

Uncertain

Verify: the specific company-attributed anecdotes below (Pinterest’s ~5 min → ~60 s Karpenter provisioning figure, Airbnb’s 2000-node PodTopologySpread benchmark, Spotify pinning percentageOfNodesToScore to 20%, OpenAI’s per-workload profiles). Reason: these were carried from the note’s original draft and were not re-confirmed against primary engineering write-ups during this verification pass; the mechanisms they illustrate (Karpenter instance-fit provisioning, PodTopologySpread O(pods) cost, lowering percentageOfNodesToScore for latency, multi-profile scheduling) are all real and independently documented, but the exact numbers and attributions are not pinned to a fetched source. To resolve: locate the originating blog post / talk for each before quoting the figures externally. uncertain

  • Pinterest’s Karpenter migration (2023): replaced the Cluster Autoscaler with Karpenter, cutting node-provisioning time from ~5 minutes to ~60 seconds for bursty workloads, primarily by letting Karpenter choose instance types that exactly fit pending Pods rather than scaling existing node groups uniformly.
  • Airbnb’s scheduler benchmarks at 2000+ nodes pointed to PodTopologySpread as the dominant Filter+Score cost — every Pod scheduling decision iterates over every other Pod in the matching topology key. Mitigations: use whenUnsatisfiable: ScheduleAnyway for soft spread instead of hard, scope spread to small topology keys (zone, not host), and use minDomains cautiously.
  • Spotify’s Kubernetes scheduling at scale: documented in 2023 that beyond ~1000 nodes the default percentageOfNodesToScore formula produced unstable placement; they pinned it to 20% and accepted slightly suboptimal placement in exchange for predictable latency.
  • OpenAI runs distinct scheduler profiles per workload class (training, inference, batch) inside a single scheduler process via the multi-profile feature, since each class benefits from a different score-weight combination.
  • Recommended monitoring: scheduler_pending_pods (gauge by queue), scheduler_pod_scheduling_attempts (counter), scheduler_e2e_scheduling_duration_seconds (histogram), scheduler_scheduling_algorithm_duration_seconds (the cycle proper), and leader_election_master_status (which replica is active).

A note on version drift: the default-enabled plugin set can shift across minor versions, so the authoritative source is always the version-specific scheduler config reference or kube-scheduler --help. The list above (the eight Score plugins, the PrioritySort QueueSort, the Filter set led by NodeResourcesFit/NodeAffinity/PodTopologySpread/TaintToleration/VolumeBinding, DefaultPreemption at PostFilter, and DefaultBinder at Bind) reflects the current default profile as of Kubernetes ~1.36 (May 2026).

The QueueingHint mechanism, gated by SchedulerQueueingHints, graduated to GA / stable in Kubernetes 1.34 and is now locked to enabled-by-default — the feature gate is no longer needed and cannot be disabled (KEP-4247 GA, website commit, Feature Gates reference). A QueueingHint is a per-plugin callback invoked when a cluster event fires (a Node added, a Pod deleted, a PVC bound) that decides whether this specific event could plausibly make a given unschedulable Pod schedulable; if not, the Pod stays parked instead of being needlessly re-attempted. This is the difference between “wake up every unschedulable Pod on every cluster change” (the old behavior) and “wake up only the Pods an event could actually unblock.”

See Also

Graph View

Backlinks