Scheduling Framework

The Scheduling Framework (KEP-624, the scheduler’s architecture since Kubernetes 1.19) is a plugin extension-point architecture: the scheduler core is a thin sequencer, and every scheduling decision — sorting the queue, gating Pods, filtering nodes, scoring them, reserving resources, binding — is performed by named, compiled-in plugins registered at well-defined extension points (Scheduling Framework docs). The companion notes Kubernetes Scheduler Architecture and kube-scheduler describe when each phase runs in the scheduling and binding cycles; this note catalogs the extension points themselves, the interface each one is a Go contract for, the default-enabled plugin set, plugin arguments, and how you write, register, profile, and ship a custom plugin. The framework’s design payoff: a feature like topology spread or inter-pod affinity is just another plugin, added or removed without forking the scheduler, and a custom scheduler is a recompile with extra plugins rather than a from-scratch rewrite.

Mental Model

Think of the scheduler as a fixed pipeline of labeled slots; plugins are objects dropped into one or more slots. A single plugin commonly implements several interfaces — NodeResourcesFit, for example, both Filters (does the Pod fit?) and Scores (how well does it fit?) and runs PreFilter (sum the requests once) — because the same domain knowledge is useful at multiple phases. The core scheduler owns only the sequencing, the parallelism, and a per-Pod scratchpad called CycleState; it owns no scheduling policy whatsoever.

flowchart LR
    subgraph QUEUE[Queue management]
        PEQ[PreEnqueue]
        QS[QueueSort]
        QH[QueueingHint /<br/>EnqueueExtension]
    end
    subgraph SCYCLE["Scheduling cycle — serial, shares CycleState"]
        PF[PreFilter] --> FI[Filter] --> POF[PostFilter]
        FI --> PRS[PreScore] --> SC[Score] --> NS[NormalizeScore]
        NS --> RES[Reserve / Unreserve]
    end
    subgraph BCYCLE[Binding cycle — async]
        PM[Permit] --> PRB[PreBind] --> BI[Bind] --> POB[PostBind]
    end
    QUEUE --> SCYCLE --> BCYCLE

What this diagram shows and the insight to extract. The extension points fall into three groups. Queue management plugins decide which Pod is worked next and when a parked Pod becomes workable again. The scheduling cycle group runs serially per Pod, shares a CycleState scratchpad, and ends at Reserve. The binding cycle group runs asynchronously. The insight: the same plugin name can appear in many slots, and the order of plugins within a slot matters — for Filter, cheap and commonly-failing checks placed first short-circuit the expensive ones — which is why the config lets you reorder, not merely enable/disable.

Mechanical Walk-through

Each extension point is a Go interface in k8s.io/kubernetes/pkg/scheduler/framework; a plugin implements the interfaces for the points it participates in, and must implement Name() string. KEP-624 motivates the design: the pre-framework scheduler had two extension mechanisms, a hard-coded predicate/priority list (every change a core fork) and an HTTP scheduler extender (a network hop per decision, no access to the scheduler cache, JSON marshalling overhead). The framework replaced both with in-process, cache-aware, compiled plugins.

The extension points, in pipeline order (the count is not fixed — Kubernetes has historically described “fourteen” core points, but the framework keeps growing: as of v1.36 two new points, PlacementGenerate and PlacementScore, were added for PodGroup/gang scheduling — see Gang and PodGroup scheduling below):

Queue management

• PreEnqueue — gates a Pod’s entry into the activeQ. A non-Success result parks the Pod in the unschedulableQ with no Unschedulable condition. The built-in SchedulingGates plugin implements this: a Pod with unsatisfied spec.schedulingGates is held until an external actor clears the gates (use: a pipeline that must provision a quota object before the Pod may schedule).
• QueueSort — provides the Less(podInfoA, podInfoB) comparator that orders the activeQ heap. Exactly one QueueSort plugin may be enabled; the default PrioritySort orders by priority descending then creation time. A heap needs one total order, hence the singleton rule — two enabled QueueSort plugins is a startup error.
• EnqueueExtension / QueueingHint — a callback a plugin registers so the queueing subsystem can ask, per cluster event, “would this event plausibly make the parked Pod schedulable?” The hint returns one of exactly two values: Queue (“this event may make the Pod schedulable” → move it to the active or backoff queue) or QueueSkip (“this event has no impact on this Pod’s schedulability” → leave it parked). An earlier design draft proposed a richer QueueImmediately/QueueAfterBackoff/QueueSkip enum, but it was explicitly rejected in favor of the two-value Queue/QueueSkip plus the backoff machinery deciding which queue (KEP-4247). Plugins implementing PreEnqueue, PreFilter, Filter, Reserve, or Permit should implement this so requeueing is precise — needlessly broad event subscriptions cause Pods to thrash through the queue on irrelevant cluster changes. The mechanism (SchedulerQueueingHints, KEP-4247) went alpha in 1.26, beta in 1.28 (enabled-by-default from 1.32), and graduated to stable/GA in Kubernetes 1.34, where it is always on; in 1.34+ every in-tree plugin ships a QueueingHint (scheduling-framework docs; GA commit).

Scheduling cycle (serial)

• PreFilter — pre-compute state shared by all Filter calls and store it in CycleState (e.g. sum the Pod’s resource requests once; build the inter-pod-affinity term index). May declare the Pod unschedulable and abort the whole cycle. PreFilterExtensions (AddPod/RemovePod) let preemption incrementally adjust this state while simulating victim eviction.
• Filter — the predicate: per node, decide feasible / infeasible. Run in parallel across nodes; per node, plugins run until one returns Unschedulable.
• PostFilter — runs only when Filter yielded zero feasible nodes. The default DefaultPreemption plugin attempts preemption here (see Pod Priority and Preemption); a successful PostFilter may set status.nominatedNodeName.
• PreScore — per-Pod-once preparation for Score; produces sharable CycleState data. It is not a reject-the-Pod-vs-a-node decision (that is Filter’s job), but it is not purely informational either: if a PreScore plugin returns an error, the scheduling cycle is aborted (scheduling-framework docs). It runs after Filter, only over the feasible nodes.
• Score — the priority: per feasible node, return an int64 rank. Score plugins run in parallel across nodes; an error from any Score plugin aborts the cycle.
• NormalizeScore — a plugin reshapes its own scores across all nodes before the weighted sum (e.g. invert utilization so “emptier scores higher”, or clamp to the score range). It is called once per plugin per scheduling cycle with that same plugin’s Score results, and is implemented via the plugin’s ScoreExtensions() method (a plugin that does not need normalization returns nil). An error from NormalizeScore aborts the cycle.
• Reserve (with its paired Unreserve) — stateful plugins mark node resources reserved for this Pod; Unreserve releases them idempotently and cannot fail if a later phase aborts. Reserve is the last scheduling-cycle point, protected by the serial guarantee.

Binding cycle (asynchronous)

• Permit — returns Approve, Deny, or Wait (timeout-bounded). Wait is the gang-scheduling hook — hold a Pod until a sibling condition is met.
• PreBind — work that must finish before binding; e.g. VolumeBinding waits here for dynamic PVC provisioning.
• Bind — perform the actual bind. Plugins run in order until one handles it (returns non-Skip); the default DefaultBinder writes the Binding subresource. At least one Bind plugin is required.
• PostBind — informational cleanup after a successful bind.

A plugin returning an error or Unschedulable from any cycle phase aborts that cycle; the Pod is re-queued and Unreserve unwinds reservations.

The default-enabled plugin set (scheduler config reference). Each plugin and the points it spans:

Plugin	Extension points it occupies	Job
PrioritySort	QueueSort	priority-descending heap order
SchedulingGates	PreEnqueue	hold Pods with unsatisfied schedulingGates
NodeName	Filter	match spec.nodeName if set
NodeUnschedulable	Filter	reject nodes marked unschedulable (cordoned)
NodePorts	PreFilter, Filter	reject nodes with hostPort conflicts
NodeAffinity	Filter, PreScore, Score	nodeSelector + node affinity (see Node Affinity)
NodeResourcesFit	PreFilter, Filter, Score	CPU/memory/extended-resource fit and scoring
NodeResourcesBalancedAllocation	Score	favor nodes whose CPU/mem usage is balanced
ImageLocality	Score	favor nodes that already cache the Pod’s images
TaintToleration	Filter, PreScore, Score	taints/tolerations (Score weight ×3)
VolumeBinding	PreFilter, Filter, Reserve, PreBind, Score*	PVC binding; Score gated on StorageCapacityScoring
VolumeRestrictions	Filter	enforce single-writer volume rules
VolumeZone	Filter	keep zonal-volume Pods in the volume’s zone
NodeVolumeLimits (+ EBSLimits/GCEPDLimits/AzureDiskLimits)	Filter	per-node attached-volume count limits
PodTopologySpread	PreFilter, Filter, PreScore, Score	even spread across topology domains
InterPodAffinity	PreFilter, Filter, PreScore, Score	pod (anti-)affinity (see Pod Affinity and Anti-Affinity)
DefaultPreemption	PostFilter	evict lower-priority Pods to make room
DefaultBinder	Bind	write the Binding subresource

Notice the recurring names — NodeResourcesFit, NodeAffinity, PodTopologySpread, InterPodAffinity, TaintToleration each span PreFilter/Filter/PreScore/Score. One body of domain logic, many slots. (NodeAffinity is enabled at filter and score by default; it also implements preScore, reachable via multiPoint.) There are no default-enabled Permit or PostBind plugins; those slots exist purely for custom plugins (gang scheduling, audit hooks) — though as of v1.36 gang scheduling is moving in-tree through a separate PodGroup mechanism rather than the per-Pod Permit point (see below). As of v1.36 the default set also wires TopologyPlacement into the new placementGenerate point and PodGroupPodsCount plus NodeResourcesFit into placementScore, both gated behind the alpha TopologyAwareWorkloadScheduling feature gate (scheduler config reference).

Gang and PodGroup Scheduling (v1.36)

The per-Pod pipeline above has a structural blind spot: it schedules one Pod at a time, so a workload that is only useful if all its Pods run together (a distributed-training job, an MPI job, a Spark executor set) can deadlock — the scheduler places half the gang, runs out of room, and the placed Pods squat on resources the rest of the gang needs, with no Pod able to make progress. Historically the workaround was an out-of-tree Permit-point plugin (Coscheduling in kubernetes-sigs/scheduler-plugins, or Volcano/YuniKorn) that holds each Pod at Wait until the whole gang is reservable.

Kubernetes v1.36 introduced a first-class, in-tree answer as alpha: a PodGroup scheduling cycle alongside the per-Pod one, plus gang scheduling semantics, all behind the TopologyAwareWorkloadScheduling (and related gang) feature gates (Kubernetes v1.36 blog; PodGroup scheduling docs).

Uncertain

Verify: the exact feature-gate name(s) governing the PodGroup cycle, gang scheduling, and topology-aware workload scheduling, and whether they are one gate or several. Reason: the v1.36 blog and config reference name TopologyAwareWorkloadScheduling for the placement points, but gang minCount may sit behind a separate gate, and these are brand-new alpha APIs (scheduling.k8s.io/v1alpha2) likely to churn before beta. To resolve: check the v1.36 kube-scheduler feature-gate list and KEP-4671 once stabilized. uncertain

The mechanism plugs into the framework via two new extension points rather than overloading the per-Pod points:

• PlacementGenerate — plugins propose candidate placements: whole sets of nodes onto which the entire PodGroup could fit at once. The default TopologyPlacement plugin generates these from the group’s topology constraints.
• PlacementScore — plugins score each proposed placement so the scheduler picks the best one for the group as a unit (rather than scoring nodes Pod-by-Pod). The default PodGroupPodsCount plugin and NodeResourcesFit (which always uses MostAllocated packing for groups) participate here.

A gang is declared with schedulingPolicy.gang.minCount on a PodGroup (the runtime-state object) or its Workload template; the scheduler treats the group atomically — if it cannot place minCount Pods simultaneously, it places none, eliminating the partial-placement deadlock. Pods join a group via a spec.schedulingGroup.podGroupName reference. The payoff for this note’s framing: gang scheduling is becoming another set of plugins at new points, the same extensibility story KEP-624 set up, rather than a fork — but it is alpha and the API shape (scheduling.k8s.io/v1alpha2) will change.

Configuration / API Surface

A plugin is a Go type implementing one or more framework interfaces. A minimal Score plugin:

// PremiumTier rewards nodes carrying a "tier=premium" label.
type PremiumTier struct{ handle framework.Handle }
 
func (p *PremiumTier) Name() string { return "PremiumTier" }
 
func (p *PremiumTier) Score(ctx context.Context, state *framework.CycleState,
    pod *v1.Pod, nodeName string) (int64, *framework.Status) {
    node, err := p.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
    if err != nil { return 0, framework.AsStatus(err) }
    if node.Node().Labels["tier"] == "premium" {
        return framework.MaxNodeScore, framework.NewStatus(framework.Success)  // 100
    }
    return framework.MinNodeScore, framework.NewStatus(framework.Success)      // 0
}
 
func (p *PremiumTier) ScoreExtensions() framework.ScoreExtensions { return nil }
 
// Factory registered into the scheduler binary's plugin registry.
func NewPremiumTier(_ context.Context, _ runtime.Object,
    h framework.Handle) (framework.Plugin, error) {
    return &PremiumTier{handle: h}, nil
}

The framework.Handle is the plugin’s window onto the scheduler: SnapshotSharedLister() exposes the consistent in-memory snapshot of nodes and Pods taken at the start of the cycle (a plugin must never read the live API server — that is slow and racy), plus client access, event recording, and the parallelizer. The factory is wired in by building a new scheduler binary that embeds the stock scheduler plus your plugin:

command := app.NewSchedulerCommand(
    app.WithPlugin("PremiumTier", NewPremiumTier))

Then the plugin is enabled per profile in the KubeSchedulerConfiguration:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler        # the implicit default profile
    plugins:
      score:
        enabled:
          - name: PremiumTier
            weight: 3                        # multiplies this plugin's normalized score
      multiPoint:
        enabled:
          - name: NodeResourcesFit           # enable across every point the plugin supports
  - schedulerName: dense-packer             # a SECOND profile in the SAME binary
    plugins:
      score:
        disabled:
          - name: "*"                        # turn off every default Score plugin
        enabled:
          - name: NodeResourcesFit
            weight: 1
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated              # bin-pack tightly
            resources:
              - name: cpu
                weight: 1
              - name: memory
                weight: 1

Line-by-line. Each entry in profiles is an independent scheduler personality identified by schedulerName; a Pod selects one via spec.schedulerName (no value → default-scheduler). All profiles run inside one scheduler process and share one informer cache — far cheaper than separate Deployments. plugins.score.enabled adds a plugin with a weight that scales its normalized contribution to the per-node weighted sum. multiPoint is a config-only convenience point: enabling a plugin there enables it at every extension point it implements, instead of listing filter, score, preScore separately. disabled with name: "*" strips every default plugin at that point so you start clean. pluginConfig passes plugin-specific arguments — here NodeResourcesFit’s scoringStrategy switches the dense-packer profile to MostAllocated (pack) versus the default LeastAllocated (spread); other plugins take args too (e.g. PodTopologySpread.defaultConstraints, InterPodAffinity.hardPodAffinityWeight, DefaultPreemption.minCandidateNodesPercentage).

Failure Modes

• Two QueueSort plugins enabled. The scheduler refuses to start — the activeQ heap requires exactly one comparator. Config validation rejects it at boot with a clear error.
• Custom plugin not in the registry. Referencing a plugin name in config that the binary was not compiled with fails startup with “plugin not found.” Plugins are compiled-in, not dynamically loaded — a config-only enable of a plugin the binary lacks is impossible. This is the single most common custom-scheduler mistake.
• Score plugin returns out-of-range values. A Score implementation returning a value outside [framework.MinNodeScore, framework.MaxNodeScore] (0–100) without a NormalizeScore that rescales it causes the framework to reject the cycle. Implement ScoreExtensions().NormalizeScore whenever raw scores can exceed the range.
• Slow custom Filter/Score. A plugin doing a network call or expensive computation per node multiplies by node count and per cycle, tanking scheduler_plugin_execution_duration_seconds. Plugins must be fast and local; expensive work belongs in PreFilter/PreScore (run once per Pod) or PreBind (async cycle).
• Permit Wait with no timeout discipline. A gang-scheduling plugin that Waits forever holds binding cycles open and leaks goroutines. Always bound Wait with a timeout and a deny-on-timeout path.
• Reading the live API in a plugin. Bypassing SnapshotSharedLister() to query the API server directly is both slow and racy — the snapshot is the contract. Plugins that do this produce nondeterministic placement.

Alternatives and When to Choose Them

• Stock scheduler, default plugins — correct for almost every cluster. Reach for custom plugins only with a concrete placement requirement the defaults cannot express.
• Custom in-tree plugin (this note) — the supported, performant way to add scheduling logic; you get the optimized snapshot and the parallelism for free. Cost: you maintain a custom scheduler binary and its release cadence.
• Scheduler extender (legacy) — the pre-framework HTTP-webhook mechanism for Filter/Prioritize/Bind. Still supported but slow (a network hop per decision), cache-blind, and limited; the framework superseded it. Use only when you genuinely cannot recompile the scheduler.
• Fully custom out-of-tree scheduler — Volcano, YuniKorn, Kueue. They embed the framework and ship a large plugin set for batch/HPC/job-queueing. See Custom Schedulers.

Production Notes

• Volcano and YuniKorn are both built on the Scheduling Framework — production proof the extension model scales to batch schedulers without forking the core. Volcano adds gang scheduling, fair-share, and queue plugins for ML/HPC; YuniKorn adds hierarchical resource queues. Kueue takes a different shape — it is a job-level admission/queueing controller that gates Pods before the default scheduler sees them, using schedulingGates plus the Permit point, rather than replacing the scheduler.
• kubernetes-sigs/scheduler-plugins is the SIG-Scheduling-maintained out-of-tree plugin collection — the canonical place to read production-grade example plugins (Coscheduling/gang scheduling at the Permit point, CapacityScheduling, NodeResourcesAllocatable, Trimaran load-aware scoring). It ships as a ready-to-deploy scheduler binary as well as importable plugin packages.
• GPU / topology-aware scheduling at NVIDIA and major ML platforms is typically a custom Score+Filter plugin pair that reads node device topology — the canonical “the defaults can’t express this” case, increasingly served by the in-tree DynamicResources (DRA) plugin instead.
• KEP-624 design history records the framework as a deliberate consolidation: it killed the predicate/priority core fork and the HTTP extender latency tax in one architecture, which is why every new scheduling feature since 1.19 ships as a plugin.

As-of caveat (verified against the v1.36 scheduler config reference, 2026-05). The default-enabled set above is current for kubescheduler.config.k8s.io/v1, but it shifts across minor versions, so pin it per cluster: plugins get added (the v1.36 TopologyPlacement and PodGroupPodsCount for PodGroup scheduling), cloud-specific volume-limit plugins (EBSLimits/GCEPDLimits/AzureDiskLimits) are being consolidated into NodeVolumeLimits, and the DRA DynamicResources plugin gains points as Dynamic Resource Allocation matures. VolumeBinding’s Score point is enabled only when the StorageCapacityScoring feature gate is on. The authoritative per-version source is the config reference for that minor, or kube-scheduler --help.

See Also

• Kubernetes Scheduler Architecture — the cycle structure these plugins slot into
• kube-scheduler — the component-level overview
• Custom Schedulers — out-of-tree and multi-scheduler deployment
• Gang Scheduling — the v1.36 in-tree PodGroup cycle and PlacementGenerate/PlacementScore points
• Node Selector — a degenerate input handled by the NodeAffinity plugin
• Node Affinity — implemented by the NodeAffinity plugin
• Pod Affinity and Anti-Affinity — implemented by the InterPodAffinity plugin
• Topology Spread Constraints — implemented by the PodTopologySpread plugin
• Taints and Tolerations — implemented by the TaintToleration plugin
• Pod Priority and Preemption — the DefaultPreemption PostFilter plugin
• Resource Requests and Limits — input to the NodeResourcesFit plugin
• Kubernetes MOC — parent index