Kubernetes GPU Scheduling
Kubernetes coordinates GPU discovery, scheduling and container access through several components. Understanding their separate roles makes allocation failures easier to diagnose.
TL;DR: A device plugin can advertise GPU resources to the kubelet; the scheduler places a pod against available resources; the node runtime makes allocated devices accessible. More demanding workloads need policies for group admission, topology and sharing. Check which capabilities the installed device driver and scheduler actually provide.
The default path, layer by layer
Five parts take part in placing a GPU job. A node is one machine in the cluster. A pod is the unit Kubernetes schedules: one or more containers that run together on a node. The scheduler is the control-plane component that picks a node for each new pod. The kubelet is the agent on every node that actually starts pods there. A device plugin is a small program on each GPU node that tells the kubelet which GPUs exist and are healthy, so they can be counted like CPU and memory.
This shows the device-plugin path for a pod requesting four whole GPUs.
Each layer has one job and one way to fail.
The device plugin is a DaemonSet that enumerates GPUs on its node and registers them with the kubelet as an extended resource. It also decides the allocation policy: which physical devices a request of 4 gets (it can prefer devices on the same NVLink or PCIe switch if told to), and whether GPUs are exposed whole, as MIG slices, or time-sliced into several virtual ones. When it crashes or is not installed, the node advertises zero GPUs and every GPU pod stays Pending with "Insufficient nvidia.com/gpu": one possible cause to check alongside exhausted capacity and unhealthy devices.
The scheduler treats the resource as an integer. It does not know that 4 GPUs on one NVSwitch beat 2 + 2 across sockets, or that a job of 32 pods needs all 32 placed together or none. Those are the two gaps the rest of this track fills: topology-aware scheduling and gang scheduling.
The container toolkit does the plumbing: device nodes, the driver's user-space libraries (which must match the host driver version, so the container's CUDA runtime is forward-compatible only within limits), and the environment variable. A container that "cannot find the GPU" needs checks of device allocation, runtime configuration, permissions and driver compatibility.
Node Feature Discovery labels nodes with what they have (GPU model, driver version, NVLink presence, MIG mode), so pods can select them with node affinity: nvidia.com/gpu.product: H100-SXM5-80GB. Without it, a job that needs H100s can land on an node that also advertises nvidia.com/gpu.
The GPU Operator packages the lot: it installs the driver as a container (or uses the host's), the toolkit, the device plugin, NFD, the DCGM exporter for metrics and a validator that checks each piece. On a managed cloud it is usually preinstalled; on bare metal it is the first thing a platform engineer deploys, and its version matrix (driver, CUDA, toolkit, plugin) is the compatibility table to keep.
What the integer cannot express
| Need | Why the integer fails | What handles it |
|---|---|---|
| a specific GPU model | nvidia.com/gpu is one count for A100s and H100s alike | NFD labels + node affinity, or DRA attributes |
| a fraction of a GPU | the count is whole numbers | MIG slices exposed as their own resource, or time-slicing, or DRA sharing (MIG, MPS and time-slicing) |
| GPUs on the same NVLink domain | the scheduler does not see topology | plugin allocation policy for intra-node; a topology-aware scheduler for multi-node |
| all-or-nothing for a multi-pod job | pods are scheduled one at a time | gang scheduling (Kueue, Volcano, KAI) |
| fair sharing between teams | first come, first served | queues and quotas in the batch scheduler |
| a NIC beside the GPU (RDMA) | separate resource, separately counted | the network operator's device plugin plus co-allocation, or DRA |
Dynamic Resource Allocation
DRA is the Kubernetes answer to the integer's limits, and it reached general availability in the 1.34 line (2025) with the NVIDIA DRA driver shipping alongside the device plugin. Instead of a count, a pod references a ResourceClaim, and the claim describes what it wants in terms of device attributes ("an NVIDIA GPU with at least 80 GB", "a MIG 3g.40gb slice", "two GPUs on the same NVLink domain"). The driver on the node publishes its devices with attributes in a ResourceSlice; the scheduler matches claims to slices with real selectors. Sharing (several pods using one claim) and partitioning (a driver that carves MIG on demand) become first-class.
As of 2026 the two mechanisms coexist. Simple workloads and most managed clouds still use the device plugin's count because it is everywhere and every chart understands it; clusters that need mixed models, sharing or topology are moving to DRA. An interviewer who asks "device plugin or DRA?" wants the reason (attributes, sharing, co-allocation) rather than the version number.
Working it in the room
Trace a pod’s GPU request through discovery, placement, allocation and runtime access. Identify the evidence at each boundary when it fails. Then extend the example to a multi-node job or a shared device, explaining what extra coordination and isolation are required.
What to remember
- Four layers: device plugin (advertises the count), scheduler (subtracts), kubelet (allocates devices), container toolkit (mounts them). Inspect each layer: insufficient resources can mean unavailable capacity, and missing device access can have several runtime causes.
- The GPU Operator installs driver, toolkit, plugin, NFD and DCGM as one chart; its version matrix is the compatibility table.
- The integer cannot express model, fraction, topology, all-or-nothing or fairness. NFD labels, MIG, DRA and a batch scheduler each fix one.
- DRA (GA in 2025) replaces the count with attribute-based claims: model, memory, sharing, MIG, co-allocation.
- Multi-node training needs gang scheduling on top; ordinary pod-by-pod placement can leave a distributed job only partially allocated.