Kubernetes GPU Scheduling
Kubernetes knows nothing about GPUs until something tells it. The NVIDIA device plugin advertises each node's GPUs as a countable resource, the scheduler matches a pod's request to a node with enough of them, and the container runtime wires the device in. That model is enough for one job per GPU and breaks the moment you need sharing, topology or multi-node placement, which is where Dynamic Resource Allocation, the GPU Operator and the batch schedulers come in. Knowing which layer does what is the platform interview's opening question.
TL;DR: The device plugin runs on every GPU node, reports
nvidia.com/gpu: 8to the kubelet, and the scheduler treats that as an integer it can subtract from. A pod asks fornvidia.com/gpu: 4, the scheduler finds a node with 4 free, the kubelet and the container toolkit mount/dev/nvidia*and the driver libraries into the container, andCUDA_VISIBLE_DEVICESis set. The GPU Operator installs all of that (driver, toolkit, plugin, DCGM exporter, node feature discovery) as a Helm chart. Dynamic Resource Allocation, GA in Kubernetes as of 2026, replaces the integer with a structured claim so a pod can ask for a GPU with specific attributes, share one, or take a MIG slice. Multi-node training and fair sharing need a scheduler on top: Kueue, Volcano, or KAI.
The default path, layer by layer
Each layer has one job and one way to fail.
The device plugin is a DaemonSet that enumerates GPUs on its node and registers them with the kubelet as an extended resource. It also decides the allocation policy: which physical devices a request of 4 gets (it can prefer devices on the same NVLink or PCIe switch if told to), and whether GPUs are exposed whole, as MIG slices, or time-sliced into several virtual ones. When it crashes or is not installed, the node advertises zero GPUs and every GPU pod stays Pending with "Insufficient nvidia.com/gpu": the single most common ticket on a new cluster.
The scheduler treats the resource as an integer. It does not know that 4 GPUs on one NVSwitch beat 2 + 2 across sockets, or that a job of 32 pods needs all 32 placed together or none. Those are the two gaps the rest of this track fills: topology-aware scheduling and gang scheduling.
The container toolkit does the plumbing: device nodes, the driver's user-space libraries (which must match the host driver version, so the container's CUDA runtime is forward-compatible only within limits), and the environment variable. A container that "cannot find the GPU" almost always has a toolkit or a driver-version problem, not a scheduling one.
Node Feature Discovery labels nodes with what they have (GPU model, driver version, NVLink presence, MIG mode), so pods can select them with node affinity: nvidia.com/gpu.product: H100-SXM5-80GB. Without it, a job that needs H100s can land on an A100 node that also advertises nvidia.com/gpu.
The GPU Operator packages the lot: it installs the driver as a container (or uses the host's), the toolkit, the device plugin, NFD, the DCGM exporter for metrics and a validator that checks each piece. On a managed cloud it is usually preinstalled; on bare metal it is the first thing a platform engineer deploys, and its version matrix (driver, CUDA, toolkit, plugin) is the compatibility table to keep.
What the integer cannot express
| Need | Why the integer fails | What handles it |
|---|---|---|
| a specific GPU model | nvidia.com/gpu is one count for A100s and H100s alike | NFD labels + node affinity, or DRA attributes |
| a fraction of a GPU | the count is whole numbers | MIG slices exposed as their own resource, or time-slicing, or DRA sharing (MIG, MPS and time-slicing) |
| GPUs on the same NVLink domain | the scheduler does not see topology | plugin allocation policy for intra-node; a topology-aware scheduler for multi-node |
| all-or-nothing for a multi-pod job | pods are scheduled one at a time | gang scheduling (Kueue, Volcano, KAI) |
| fair sharing between teams | first come, first served | queues and quotas in the batch scheduler |
| a NIC beside the GPU (RDMA) | separate resource, separately counted | the network operator's device plugin plus co-allocation, or DRA |
Dynamic Resource Allocation
DRA is the Kubernetes answer to the integer's limits, and it reached general availability in the 1.34 line (2025) with the NVIDIA DRA driver shipping alongside the device plugin. Instead of a count, a pod references a ResourceClaim, and the claim describes what it wants in terms of device attributes ("an NVIDIA GPU with at least 80 GB", "a MIG 3g.40gb slice", "two GPUs on the same NVLink domain"). The driver on the node publishes its devices with attributes in a ResourceSlice; the scheduler matches claims to slices with real selectors. Sharing (several pods using one claim) and partitioning (a driver that carves MIG on demand) become first-class.
As of 2026 the two mechanisms coexist. Simple workloads and most managed clouds still use the device plugin's count because it is everywhere and every chart understands it; clusters that need mixed models, sharing or topology are moving to DRA. An interviewer who asks "device plugin or DRA?" wants the reason (attributes, sharing, co-allocation) rather than the version number.
Working it in the room
The question is usually "walk me through what happens when a pod asks for 4 GPUs," and the scored answer names the four layers in order (device plugin advertises, scheduler subtracts, kubelet allocates, toolkit mounts) with the failure each produces. The follow-up held back is "and for a 32-GPU training job?", which opens gang scheduling and topology; the second follow-up is "and if two jobs each need half a GPU?", which opens MIG and DRA. The answer that sounds right and fails is "Kubernetes schedules GPUs like CPUs": it counts them like CPUs and understands them far less.
What to remember
- Four layers: device plugin (advertises the count), scheduler (subtracts), kubelet (allocates devices), container toolkit (mounts them). "Insufficient nvidia.com/gpu" is the plugin; "cannot find GPU" is the toolkit.
- The GPU Operator installs driver, toolkit, plugin, NFD and DCGM as one chart; its version matrix is the compatibility table.
- The integer cannot express model, fraction, topology, all-or-nothing or fairness. NFD labels, MIG, DRA and a batch scheduler each fix one.
- DRA (GA in 2025) replaces the count with attribute-based claims: model, memory, sharing, MIG, co-allocation.
- Multi-node training needs gang scheduling on top; Kubernetes alone will place half a job and deadlock the rest.