vault

Batch Workloads on Kubernetes

14 min read

Batch Workloads on Kubernetes

A batch workload is work that runs to completion rather than serving requests indefinitely — a data-processing pipeline, a Monte-Carlo simulation, an HPC solver, a distributed machine-learning training run. Kubernetes ships a built-in batch primitive, the Job (scheduled by CronJob), and for simple batch — one migration, a nightly report, an embarrassingly-parallel image transform — the Job is entirely sufficient. But the moment batch goes to scale — many teams sharing one cluster, thousands of jobs competing for finite GPUs, a distributed training job whose 64 workers must all start together or not at all — three capabilities the stock Job and the default kube-scheduler simply do not have become mandatory: queueing (admit only as many jobs as the cluster can actually run, instead of flooding it with pending Pods), fair sharing and quotas (carve cluster capacity among teams, let idle capacity be borrowed and reclaimed), and gang scheduling (all-or-nothing placement for jobs whose Pods are useless individually). An ecosystem of batch frameworks — Kueue, Volcano, JobSet, Apache YuniKorn — exists precisely to supply these. This note is about that gap and how the ecosystem closes it.

Mental Model

The default Kubernetes scheduler is Pod-by-Pod and admission-blind. It pulls one unscheduled Pod at a time, finds it a node, binds it, repeats — with no notion of “job,” no notion of “this job’s 64 Pods are a unit,” and no notion of “this team has already used its quota.” Batch frameworks insert two new layers above the scheduler.

flowchart TB
    subgraph TEAMS["Teams submit jobs"]
      J1["Job / JobSet / MPIJob / RayJob<br/>team-ml"]
      J2["Job<br/>team-analytics"]
    end
    subgraph QUEUE["Layer 1 — QUEUEING & QUOTA (Kueue)"]
      LQ["LocalQueue (namespaced)<br/>team's entry point"]
      CQ["ClusterQueue (cluster-scoped)<br/>nominal quota, borrowing rules"]
      COH["Cohort<br/>ClusterQueues that lend/borrow<br/>idle quota; fair sharing"]
      ADMIT{"Admit?<br/>quota available<br/>OR borrowable from cohort"}
    end
    subgraph GANG["Layer 2 — GANG SCHEDULING (Volcano / coscheduling)"]
      ALLOR{"All N Pods<br/>placeable at once?"}
    end
    SCHED["kube-scheduler / Volcano scheduler<br/>binds Pods to Nodes"]
    NODES[(Nodes: CPU / GPU)]

    J1 --> LQ --> CQ
    J2 --> LQ
    CQ --> COH
    CQ --> ADMIT
    ADMIT -- "no: stay queued<br/>(suspended, zero Pods)" --> CQ
    ADMIT -- "yes: un-suspend the Job" --> GANG
    ALLOR -- "no: hold, retry" --> GANG
    ALLOR -- "yes: bind the whole gang" --> SCHED
    SCHED --> NODES

What this diagram shows. A job submitted by a team does not go straight to the scheduler. It first hits the queueing layer (Kueue): the job is suspended (zero Pods created) and parked in a LocalQueue, which points at a ClusterQueue holding a quota budget. Kueue admits the job only when the ClusterQueue has nominal quota — or can borrow idle quota from sibling queues in its cohort. Only an admitted job is un-suspended and allowed to create Pods. Those Pods then pass through the gang-scheduling layer (Volcano or the coscheduling plugin), which holds them until all of a job’s Pods can be placed simultaneously, then binds the whole gang. The insight to extract is that the default scheduler is a good Pod placer but a poor batch admission controller — it has no concept of “too many jobs” or “this job is incomplete.” The batch ecosystem adds the missing admission and all-or-nothing semantics above the scheduler, leaving the scheduler to do what it is good at.

Mechanical Walk-through

Why stock Job + kube-scheduler is not enough at scale

The Job API is genuinely capable for a single batch task — parallelism, completions, backoffLimit, Indexed completion mode (GA in Kubernetes 1.24, per the Jobs concept docs), backoffLimitPerIndex (a per-index retry budget for indexed jobs), podFailurePolicy (which exit codes or Pod conditions should fail-fast the whole job versus be retried), successPolicy (declare a Job successful before all completions finish — useful for leader-elected workloads where one Pod’s success ends the run), and the managedBy field (GA in 1.35) that lets the built-in Job controller step aside so an external controller — Kueue, JobSet, an in-house operator — owns reconciliation. This last point matters here: managedBy is the contract that lets Kueue and JobSet coexist cleanly with the upstream batch primitive instead of fighting it. What the Job API does not address:

  1. No queueing / admission control. If you kubectl apply 500 Jobs and the cluster can only run 50 at once, the Job controller dutifully creates Pods for all 500. The 450 that don’t fit pile up as Pending Pods, hammering the scheduler, cluster autoscaler, and API server. There is no “wait your turn” — every Job races for capacity the instant it is created.

  2. No cross-team fair sharing or quota. ResourceQuota caps a namespace’s total resource use, but it is a hard ceiling, not a fair share: it cannot say “team A gets 100 GPUs nominally, but may borrow team B’s idle GPUs, and must give them back when B needs them.” Batch clusters need lending and reclaim, not just ceilings.

  3. No gang scheduling. This is the deepest gap. A distributed training job’s 64 worker Pods are useless individually — worker 0 sitting idle, holding 8 GPUs, waiting for workers 1–63 that can’t be scheduled is pure waste, and if every job behaves this way the cluster deadlocks: every job holds some GPUs and no job has enough to run. The default scheduler binds Pods independently; it cannot express “place all 64 or place none.” This is the canonical reason a batch/HPC/ML cluster needs more than kube-scheduler.

Kueue — the SIG-native job-queueing system

Kueue (kueue.sigs.k8s.io) is the Kubernetes-SIG project for job-level queueing and quota. It does not schedule Pods — it controls Job admission, then lets the normal scheduler place the admitted Job’s Pods. Its core objects:

  • Workload — Kueue’s internal representation of one unit of admittable work (created automatically for each Job/JobSet/etc.). It carries the resource request and the queue it belongs to.
  • LocalQueue — a namespaced object; a team’s entry point. A team submits Jobs labelled with a LocalQueue name. The LocalQueue points at a ClusterQueue.
  • ClusterQueue — a cluster-scoped object governing a pool of resources: it defines nominal quota (the guaranteed amount), borrowing limits, and lending limits (kueue.sigs.k8s.io — ClusterQueue).
  • ResourceFlavor — represents a variation of a resource — e.g. gpu-a100 vs gpu-h100, or arm64 vs amd64 nodes. Quota is expressed per flavor, and a ResourceFlavor can carry node labels/taints that pin it to a node group.
  • Cohort — a set of ClusterQueues that can borrow each other’s unused quota. A queue may borrow from idle siblings up to its borrowingLimit, and lend its own idle quota up to its lendingLimit (kueue.sigs.k8s.io — administer quotas).

The admission flow: a Job is submitted suspended (spec.suspend: true — zero Pods created). Kueue creates a Workload, parks it in the LocalQueue/ClusterQueue, and admits it only when nominal quota is available or the cohort has borrowable idle capacity. On admission, Kueue flips spec.suspend to false and the Job controller creates Pods normally. Fair Sharing orders waiting jobs so no single tenant monopolises a shared ClusterQueue — Kueue tracks historical usage and favours the queue with the lowest accumulated consumption. Kueue has built-in integrations for batch Job, JobSet, MPIJob, RayJob, PyTorchJob, and more (github.com/kubernetes-sigs/kueue).

Volcano — the CNCF batch scheduler

Volcano (volcano.sh) takes the other approach: rather than gating admission above the default scheduler, it replaces the scheduler with one purpose-built for batch. Volcano is a CNCF Incubating project — accepted to the CNCF on 9 April 2020 and promoted to Incubating on 21 March 2022 (cncf.io/projects/volcano). It has not graduated as of this note’s date; references that call it “CNCF-graduated” are wrong, and the conflation is one of the routinely-mistaken facts in this corner of the ecosystem. Its headline capability is gang scheduling: a Volcano PodGroup (or its higher-level Job CRD) declares a minMember, and Volcano binds the group only when at least minMember Pods can be placed simultaneously — the all-or-nothing semantics distributed training and HPC require. Volcano also provides queue-based fair-share scheduling, binpack, NUMA-aware placement, task-topology scheduling, and DeviceShare for GPU sharing. It is the default scheduler under most Kubeflow, Ray-on-Kubernetes, and HPC deployments.

As of 2026-05-30 the latest releases are v1.14.2, v1.13.3, and v1.12.4, all published on 2026-05-09 as a coordinated security patch addressing an unbounded-HTTP-body DoS in the webhook server (volcano-sh/volcano releases). The project’s README describes itself as “a Kubernetes-native batch scheduling system, extending and enhancing the capabilities of the standard kube-scheduler” and notes widespread industry adoption across cloud, finance, manufacturing, and medical sectors (volcano-sh/volcano README). A March 2026 CNCF post frames the v1.14 line as Volcano “evolving into the AI-native unified scheduling platform” — i.e. positioning beyond pure batch to a single scheduler for training, inference, and traditional batch (cncf.io blog 2026-03). The exact roadmap features under that umbrella shift quickly across point releases; pin to the changelog for your installed minor version when relying on a specific capability.

JobSet — grouping Jobs as one unit

JobSet (jobset.sigs.k8s.io, kubernetes.io blog 2025-03) is a higher-level API that models a distributed workload as a group of Kubernetes Jobs managed as a unit. A distributed ML training run is rarely homogeneous — it has a leader, a set of workers, sometimes parameter servers, each needing a different Pod template. JobSet’s ReplicatedJob abstraction lets you declare each role as its own child Job, and (since JobSet v0.6.0) declare a startup order so the leader is running before workers connect. JobSet integrates with Kueue for admission control, so a JobSet is queued, fair-shared, and gang-admitted as one unit.

Apache YuniKorn — capacity scheduling

Apache YuniKorn (originally from LinkedIn) is a replacement scheduler offering hierarchical queue capacity scheduling, modelled on Hadoop YARN’s CapacityScheduler — useful for multi-tenant clusters where teams need guaranteed and elastic slices of a hierarchy of queues. It is the alternative to Volcano for organisations whose mental model is YARN-style capacity hierarchies. (See kube-scheduler for its place in the scheduler-alternatives landscape.)

Configuration / API Surface

A minimal Kueue setup — a ResourceFlavor, a ClusterQueue with GPU quota, a LocalQueue, and a Job submitted into it:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: gpu-a100                          # a named variation of "GPU"
spec:
  nodeLabels:
    gpu-type: a100                        # this flavor maps to nodes with this label
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-cq                       # cluster-scoped: a pool of capacity
spec:
  namespaceSelector: {}                   # which namespaces may use this queue
  cohort: shared-gpu-pool                 # queues in the same cohort lend/borrow
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: gpu-a100
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 64             # guaranteed 64 A100s for this queue
              borrowingLimit: 32           # may borrow up to 32 more from cohort
            - name: "cpu"
              nominalQuota: "512"
            - name: "memory"
              nominalQuota: 2Ti
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: team-ml-lq                        # namespaced: a team's entry point
  namespace: team-ml
spec:
  clusterQueue: research-cq               # points at the cluster-scoped quota pool
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-run-2026-05-16
  namespace: team-ml
  labels:
    kueue.x-k8s.io/queue-name: team-ml-lq # THE label that hands the Job to Kueue
spec:
  suspend: true                           # REQUIRED: submit suspended; Kueue un-suspends on admission
  parallelism: 8
  completions: 8
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: trainer
          image: registry.example.com/trainer:v9
          resources:
            limits:
              nvidia.com/gpu: 1            # 8 Pods × 1 GPU = 8 GPUs requested

Line-by-line, the load-bearing fields:

  • ResourceFlavor.nodeLabels — ties the abstract flavor gpu-a100 to physical nodes; quota is counted per flavor.
  • ClusterQueue.cohort — putting research-cq in cohort shared-gpu-pool lets it borrow idle GPUs from sibling queues in the same cohort, up to borrowingLimit.
  • nominalQuota: 64 — the guaranteed allocation; borrowingLimit: 32 — the elastic ceiling on top of it.
  • labels: kueue.x-k8s.io/queue-name — the single label that diverts a Job from “schedule immediately” to “Kueue-managed admission.”
  • spec.suspend: true — mandatory for Kueue-managed Jobs. The Job is created with zero Pods; Kueue flips it to false only when quota is available. Forgetting this is the #1 Kueue setup mistake — the Job runs immediately, bypassing the queue entirely.

Failure Modes

  1. Forgetting spec.suspend: true. A Job submitted without suspend: true runs immediately and is never gated by Kueue — quota and fair sharing are silently bypassed. Symptom: jobs run despite the ClusterQueue being “full.” Fix: always submit suspended (Kueue webhooks can also default this).

  2. Gang-scheduling deadlock without gang support. Without Volcano or the coscheduling plugin, ten 64-Pod training jobs each grab a partial slice of GPUs, none reaches 64, and the cluster deadlocks — every GPU held, no job runnable. Symptom: many jobs at 30–60% of their Pods running, indefinitely, zero progress. Fix: a gang scheduler that places all-or-nothing.

  3. Quota that ignores GPUs entirely. A ClusterQueue that lists CPU and memory quota but omits nvidia.com/gpu does not gate GPU consumption — GPU jobs are admitted on CPU/memory headroom alone and then thrash for GPUs. Fix: include every scarce resource (especially GPUs) in coveredResources.

  4. Two schedulers fighting over the same Pods. Running Volcano and leaving Pods on the default scheduler — or running Kueue and a separate gang scheduler with inconsistent assumptions — produces double-binding races and stuck Pods. Pick a coherent stack; set schedulerName deliberately.

  5. borrowingLimit set too high → noisy-neighbour starvation. If every queue can borrow unlimited idle quota, a burst from one team can consume the whole cluster and the reclaim (preemption to return borrowed quota) is disruptive. Tune borrowingLimit/lendingLimit and preemption policy together.

  6. JobSet startup-order mismatch. A JobSet whose workers start before the leader (no startup order configured) wedges — workers spin trying to connect to a leader that does not exist. Fix: configure the leader-worker startup order (JobSet v0.6.0+).

Alternatives and When to Choose Them

  • Stock Job / CronJob — choose for simple batch: a single migration, a nightly report, a one-off parallel transform on a cluster you control. No queueing layer needed.
  • Kueue — choose when many teams share a cluster and you need job-level queueing, quota, fair sharing, and borrowing on top of the normal scheduler. Kueue does not schedule Pods; it gates Job admission. The SIG-native, lightest-touch option.
  • Volcano — choose when you need gang scheduling and are willing to run a replacement scheduler. The default for AI/ML and HPC workloads with tightly-coupled multi-Pod jobs. CNCF Incubating.
  • JobSet — choose to group multiple Jobs (leader + workers + parameter servers) as one unit for distributed ML; compose it with Kueue for admission.
  • Apache YuniKorn — choose when your mental model is YARN-style hierarchical capacity queues and you want a single scheduler implementing them.
  • Argo Workflows / Tekton Pipelines — choose for DAG-shaped pipelines with step dependencies and parameter passing, a different problem from fleet-level job queueing (these orchestrate Pods directly; you can still queue them with Kueue).
  • Kubeflow — the ML-platform layer that composes these — Volcano for gang scheduling, the Training Operator’s CRDs, optionally Kueue for queueing.

Production Notes

  • Kueue + JobSet is Google’s published pattern for running large training jobs on GKE — Kueue for admission control and fair sharing, JobSet to compose the leader/worker/parameter-server roles, with topology-aware placement underneath (kubernetes.io blog).
  • Volcano under Kubeflow and Ray. Essentially every serious Kubeflow or Ray-on-Kubernetes deployment uses Volcano (or the coscheduling plugin) as the scheduler, because the bare kube-scheduler cannot gang-schedule a multi-worker training job (volcano.sh).
  • The deadlock is real, not theoretical. The canonical batch-cluster incident is the gang-scheduling deadlock: without all-or-nothing semantics a busy ML cluster grinds to zero throughput while every GPU sits “in use” by a partially-scheduled job. This single failure mode is why the entire gang-scheduling ecosystem exists.
  • Quota borrowing changed the economics. Kueue’s cohort borrowing turns rigid per-team ResourceQuota ceilings into an elastic shared pool — idle GPUs are lent out and reclaimed — which is how organisations push expensive GPU clusters toward high utilisation instead of fragmenting them into stranded per-team slices.

See Also

Graph View

Backlinks