vault

GPU Scheduling on Kubernetes

14 min read

GPU Scheduling on Kubernetes

Kubernetes was designed around two fungible resources — CPU and memory — both compressible-or-divisible and modelled as continuous quantities. A GPU is neither: it is a discrete, expensive, specialised device that the kernel and the default scheduler know nothing about. To run GPU workloads — deep-learning training, inference serving, scientific computing, video transcoding — Kubernetes had to grow a mechanism for a vendor to teach the cluster about hardware the core platform was never told about. The original mechanism is the device plugin framework: a vendor-supplied DaemonSet discovers GPUs on each node and advertises them to the kubelet as an extended resource (e.g. nvidia.com/gpu), which the kube-scheduler then treats as a simple countable quantity, like a quota of integers. A Pod requests limits: nvidia.com/gpu: 1 and the scheduler finds a node with a spare GPU. On top of this sit two further concerns: operationalising the whole driver-and-toolkit stack (the NVIDIA GPU Operator) and sharing one physical GPU among several Pods (time-slicing, MIG, MPS). And, since Kubernetes v1.34 (1 September 2025), a more general successor to the device-plugin model — Dynamic Resource Allocation (DRA) — has had its core framework reach general availability (stable API resource.k8s.io/v1, on by default), redefining how complex devices are requested. This note covers the whole stack.

Mental Model

A GPU enters the Kubernetes resource model the way a foreign substance enters a body — through a gatekeeper that translates it into the body’s own vocabulary. The device plugin is that gatekeeper.

flowchart TB
    subgraph NODE["A GPU node"]
      HW[("Physical GPUs<br/>2× NVIDIA H100")]
      DRV["NVIDIA driver +<br/>container toolkit"]
      DP["NVIDIA device plugin (DaemonSet Pod)<br/>discovers GPUs, registers with kubelet"]
      KUBELET["kubelet<br/>advertises nvidia.com/gpu: 2<br/>in Node.status.allocatable"]
      HW --- DRV
      DRV --- DP
      DP -- "gRPC: ListAndWatch + Allocate" --> KUBELET
    end
    SCHED["kube-scheduler<br/>NodeResourcesFit: treats<br/>nvidia.com/gpu as countable"]
    POD["Pod spec:<br/>resources.limits:<br/>  nvidia.com/gpu: 1"]
    KUBELET -- "node advertises 2 GPUs" --> SCHED
    POD -- "requests 1 GPU" --> SCHED
    SCHED -- "binds Pod to a node<br/>with a free GPU" --> KUBELET
    KUBELET -- "Allocate(): inject device,<br/>env vars, mounts into container" --> HW

What this diagram shows. Kubernetes core has no idea what a GPU is. The vendor’s device plugin — running as a DaemonSet Pod on every GPU node — discovers the hardware, then over a gRPC contract tells the local kubelet “this node has 2 of nvidia.com/gpu.” The kubelet folds that into Node.status.allocatable, and from that moment the kube-scheduler’s NodeResourcesFit plugin treats nvidia.com/gpu exactly like CPU or memory — a countable extended resource. A Pod that requests limits: nvidia.com/gpu: 1 is placed on a node with a spare count; when the Pod lands, the kubelet calls the device plugin’s Allocate() to inject the actual device files, environment variables, and library mounts into the container. The insight to extract is that GPU scheduling, in the device-plugin model, is just integer counting — the scheduler does not know an H100 from a T4, cannot say “any GPU with ≥ 40 GB memory,” cannot share a GPU between Pods. Those limitations are exactly what motivated the GPU Operator’s sharing features and, ultimately, DRA.

Mechanical Walk-through

The device plugin framework

The device plugin (kubernetes.io — device plugins) is a stable Kubernetes extension point. A vendor ships a plugin — for NVIDIA, the nvidia-device-plugin — deployed as a DaemonSet so one copy runs on every GPU node. The plugin registers with the kubelet over a gRPC socket and implements two key calls:

  • ListAndWatch — streams the list of devices on the node and their health to the kubelet. The kubelet aggregates the count into Node.status.allocatable under the extended resource name nvidia.com/gpu.
  • Allocate — called when a Pod requesting the resource is scheduled to the node; returns the device nodes, environment variables (NVIDIA_VISIBLE_DEVICES), and library mounts the container needs.

An extended resource is the generic Kubernetes mechanism for “a countable thing the core platform doesn’t know about.” It must be requested with limits equal to requests — extended resources are integers, not fractions, and cannot be overcommitted. A Pod cannot request 0.5 of a GPU in this model; the smallest unit is 1 whole GPU. This integer-counting constraint is the device-plugin model’s defining limitation: no fractional requests, no attribute-based selection (“any GPU with 80 GB”), no sharing.

The NVIDIA GPU Operator — operationalising the stack

Getting a GPU node ready is more than installing a device plugin. You need: a matching NVIDIA driver on the node, the NVIDIA container toolkit (so the runtime can inject GPUs into containers), the device plugin itself, DCGM (Data Center GPU Manager) for monitoring, GPU Feature Discovery (GFD) for node labels, and — for partitioning — a MIG manager. Installing and version-matching all of that by hand, across a fluctuating fleet, is fragile.

The NVIDIA GPU Operator (docs.nvidia.com) is an Operator Pattern controller that automates the entire stack. It manages, as a coordinated set of DaemonSets:

  • The NVIDIA driver (containerised driver install, so nodes need no pre-baked driver).
  • The container toolkit (wires the container runtime to expose GPUs).
  • The device plugin (advertises nvidia.com/gpu).
  • The DCGM exporter (GPU metrics — utilisation, temperature, memory, power — exported to Prometheus; see Prometheus on Kubernetes).
  • GPU Feature Discovery (labels nodes with GPU model, memory, compute capability — usable in Node Affinity).
  • The MIG manager (applies MIG partition layouts).

The payoff: a bare node joins the cluster, the GPU Operator detects it, installs the whole stack as Pods, and the node becomes a working GPU node with no manual driver wrangling.

Sharing a GPU — time-slicing, MIG, MPS

A single H100 is expensive; an inference Pod or a notebook may use only a fraction of it. Three sharing mechanisms exist (docs.nvidia.com — GPU sharing):

  1. Time-slicing. The administrator defines a number of replicas of a GPU; each replica can be handed to a separate Pod. Internally the GPU context-switches between the Pods’ workloads. Time-slicing provides no memory isolation and no fault isolation — one Pod can exhaust GPU memory and OOM its neighbours, and a fault in one workload can affect others. It works on any GPU generation, including older cards that do not support MIG. It is the right choice for low-stakes shared access — dev notebooks, light inference — where utilisation matters more than isolation.

  2. MIG — Multi-Instance GPU. On NVIDIA A100/H100-class GPUs, MIG partitions one physical GPU into up to seven smaller, hardware-isolated instances, each with its own dedicated memory, cache, and compute slice. Each MIG instance looks like a “mini-GPU” and provides genuine memory and fault isolation at the hardware layer. The GPU Operator’s MIG manager applies a partition profile (e.g. 1g.10gb × 7) and the device plugin then advertises the MIG instances as schedulable resources. MIG is the right choice for multi-tenant GPU sharing where one tenant must not be able to harm another. The trade-off versus time-slicing: fewer, fixed-size slices, available only on MIG-capable hardware.

  3. MPS — Multi-Process Service. NVIDIA’s mechanism for letting multiple processes share a GPU context concurrently with finer control than time-slicing’s coarse context-switch, sitting between time-slicing and MIG in the isolation/utilisation trade-off.

MIG and time-slicing can be combined — time-slice the individual MIG instances — for even higher sharing density on a single physical GPU.

Dynamic Resource Allocation (DRA) — the successor model

The device-plugin model’s “countable integers” abstraction is too crude for the modern GPU world: you cannot say “any GPU with at least 40 GB,” you cannot share a device across unrelated Pods with an admin-defined policy, you cannot express rich topology constraints. Dynamic Resource Allocation (DRA) is the more general resource.k8s.io API built to fix this.

DRA’s core objects (kubernetes.io — DRA):

  • DeviceClass — a category of devices, with CEL-based attribute filters. The admin/driver defines what kinds of device exist and how to ask for them.
  • ResourceClaim — a workload’s request for device access — “I need a device from this DeviceClass matching these selectors.” Analogous to a PersistentVolumeClaim for storage.
  • ResourceClaimTemplate — generates a per-Pod ResourceClaim automatically, the way volumeClaimTemplates generates per-replica PVCs.
  • ResourceSlice — published by the device driver; describes the devices attached to nodes, which the scheduler queries to place Pods.

A workload describes what kind of device it needs (“a GPU with ≥ 20 GB memory”) and the scheduler, together with the DRA driver, allocates the actual device — far more expressive than counting nvidia.com/gpu: 1. The full object model and lifecycle are covered in Dynamic Resource Allocation; this section’s purpose is to place DRA correctly in the GPU-scheduling story.

The core DRA framework graduated to General Availability in Kubernetes v1.34 (released 1 September 2025), with the stable API at resource.k8s.io/v1 and DRA enabled by default — adopters no longer need to flip a feature gate to use the stable API (Kubernetes v1.34 DRA blog, 2025-09-01). The “v1.35” figure that circulated in some secondary write-ups was simply wrong; the primary Kubernetes blog pins GA to v1.34. What does vary by sub-feature is the stability tier of the extensions layered on top of the GA core. As of v1.34, DeviceClass, ResourceClaim, ResourceClaimTemplate, and ResourceSlice are all GA (the four objects described above). Several enhancements were still maturing: promoted to beta were admin-access labelling (the resource.k8s.io/admin-access: "true" namespace gate that restricts privileged-device claims), the prioritized list (a workload offering the scheduler an ordered set of acceptable device alternatives), and the PodResources API reporting of DRA-allocated devices for node monitoring agents; newly alpha were extended-resource mapping (advertising DRA-managed devices under a classic extended-resource name so unmodified workloads can consume them), consumable capacity (admin-defined policies for sharing a single device across independent claims), and binding conditions (delaying Pod-to-node binding until an external resource is confirmed ready). For any cluster, confirm the per-feature tier against that release’s notes, since these advance every minor.

DRA does not instantly retire the device-plugin model — the simple nvidia.com/gpu: 1 request remains valid and widely used — but it is the strategic direction for complex device scheduling.

Configuration / API Surface

The classic device-plugin request, then a DRA-style request:

# --- Device-plugin model: GPU as a countable extended resource ---
apiVersion: v1
kind: Pod
metadata:
  name: trainer
spec:
  nodeSelector:
    gpu-type: h100                         # GFD-applied label → pick an H100 node
  containers:
    - name: trainer
      image: registry.example.com/trainer:v9
      resources:
        limits:
          nvidia.com/gpu: 1                # request 1 WHOLE GPU; limits==requests,
                                           # integer only, no fractions, no overcommit
---
# --- DRA model: request a device by ATTRIBUTE, not by count ---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: large-gpu
spec:
  spec:
    devices:
      requests:
        - name: gpu
          deviceClassName: gpu.nvidia.com # the DeviceClass the driver published
          selectors:
            - cel:
                expression: |              # CEL: pick any GPU with >= 40 GB memory
                  device.capacity['memory'].compareTo(quantity('40Gi')) >= 0
---
apiVersion: v1
kind: Pod
metadata:
  name: trainer-dra
spec:
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: large-gpu # per-Pod ResourceClaim auto-generated
  containers:
    - name: trainer
      image: registry.example.com/trainer:v9
      resources:
        claims:
          - name: gpu                      # bind this container to the claimed device

The contrast that matters: the device-plugin Pod asks for a number (nvidia.com/gpu: 1) and relies on a nodeSelector label to steer toward the right hardware; the DRA Pod asks for a device with properties (memory >= 40Gi) and lets the scheduler and driver find it. DRA’s expressiveness — attribute selection, sharing policies, topology awareness — is the whole reason it exists.

Topology / NUMA awareness applies to both: the kubelet’s Topology Manager can align a Pod’s GPU, CPU, and memory allocations to the same NUMA node so a training process is not bottlenecked shuttling data across the NUMA interconnect — important for high-throughput GPU workloads.

Failure Modes

  1. No device plugin → GPUs invisible. A node with physical GPUs but no running device plugin advertises nvidia.com/gpu: 0. GPU Pods stay Pending with Insufficient nvidia.com/gpu. Diagnose: kubectl describe node shows no GPU in Allocatable; check the device-plugin DaemonSet Pod is Running on that node.

  2. Driver / toolkit / plugin version skew. A device plugin built against a driver version the node does not have produces Pods that schedule but fail at runtime with CUDA initialisation errors. The GPU Operator exists precisely to keep these versions coherent — hand-managed stacks drift.

  3. requests ≠ limits for the extended resource. The API rejects a GPU request where requests and limits differ, or fractional values — extended resources are integers and cannot be overcommitted. Symptom: admission error. Fix: limits: nvidia.com/gpu: <integer>, no separate requests.

  4. Time-slicing OOM cross-talk. Time-slicing gives no memory isolation: a Pod that allocates all of the GPU’s VRAM OOMs its time-sharing neighbours. Symptom: unrelated GPU Pods on the same card failing with out-of-memory. Fix: use MIG for genuine isolation, or stop oversubscribing.

  5. MIG requested on a non-MIG GPU. Applying a MIG profile to a T4 or consumer card fails — only A100/H100-class GPUs support MIG. Diagnose: MIG manager logs report unsupported hardware.

  6. Stranded GPUs from poor bin-packing. A node with 8 GPUs running 8 single-GPU Pods is perfectly packed; the same node running mixed 1- and 3-GPU requests can leave isolated single GPUs that no multi-GPU job can use. The default scheduler is not GPU-topology-aware by default — fragmentation strands expensive hardware. Volcano’s binpack/DeviceShare or DRA-aware scheduling mitigate this; see Batch Workloads on Kubernetes.

  7. Gang-less multi-GPU job deadlock. A distributed training job needing 32 GPUs, scheduled Pod-by-Pod, grabs a partial slice and waits forever for the rest — and a cluster full of such jobs deadlocks. The fix is gang scheduling, covered in Batch Workloads on Kubernetes.

Alternatives and When to Choose Them

  • Device plugin (nvidia.com/gpu extended resource) — choose for the simple, common case: whole-GPU requests, one GPU per Pod, no sharing. Mature, ubiquitous, well understood.
  • NVIDIA GPU Operator — choose to operationalise the whole stack rather than hand-installing drivers. The default for any non-trivial NVIDIA GPU cluster.
  • Time-slicing — choose for low-stakes shared access (notebooks, light inference) where utilisation beats isolation and the hardware may be older.
  • MIG — choose for multi-tenant GPU sharing on A100/H100-class hardware where tenants must be hardware-isolated.
  • MPS — choose for concurrent multi-process sharing with finer control than time-slicing, where MIG’s fixed partitioning is too rigid.
  • Dynamic Resource Allocation (DRA) — choose for complex device scheduling: attribute-based selection, admin-defined sharing policies, topology-aware allocation. The strategic successor; the core framework is GA with resource.k8s.io/v1 stable and on by default since v1.34, though several enhancements remain beta/alpha — verify per-feature stability for your cluster version.
  • Cloud-managed GPU node pools (EKS/GKE/AKS GPU pools) — choose to let the cloud provider manage the GPU AMI/driver lifecycle; the GPU Operator or a vendor-baked image still advertises the resource.

Production Notes

  • The GPU Operator is the de facto standard. Hand-managing NVIDIA drivers across a fleet is a known source of version-skew incidents; the GPU Operator’s coordinated DaemonSet management is why most production NVIDIA clusters adopt it (docs.nvidia.com).
  • CERN’s shared-GPU work documented the practical trade-offs of time-slicing versus MIG for a multi-user research cluster — time-slicing maximises the number of users per GPU at the cost of isolation; MIG gives isolation at the cost of fixed slice sizes.
  • DRA is the most-watched scheduling change in years. The Kubernetes community’s framing is explicit: DRA generalises GPU/device scheduling the way the Container Storage Interface generalised storage, and its v1.34 GA is treated as a milestone for AI/ML workloads on Kubernetes (kubernetes.io blog).
  • GPU scheduling rarely stands alone. In practice it composes with Batch Workloads on Kubernetes (gang scheduling so multi-GPU jobs don’t deadlock) and Kubeflow (the ML-platform layer) — a GPU cluster is the device plugin / DRA layer plus a batch scheduler plus an ML platform.

See Also

Graph View