Walk me through what happens when a pod asks Kubernetes for four GPUs, from the manifest to the container seeing them.

TL;DR: The device plugin on each node advertises nvidia.com/gpu: 8 to the kubelet; the scheduler subtracts a pod's request from the free count on each node and picks one with at least 4; the kubelet asks the plugin for 4 device IDs; the container toolkit mounts /dev/nvidia0..3 and the driver libraries and sets CUDA_VISIBLE_DEVICES. Each layer fails in its own way ("Insufficient nvidia.com/gpu" is the plugin, "no CUDA-capable device" is the toolkit), and because the scheduler only sees an integer, it cannot express model, fraction, topology or all-or-nothing, which is what Dynamic Resource Allocation adds.

How to approach it

Ask whether the cluster uses the device plugin's integer resource or a DRA driver, because the answer differs in the middle. Then walk the four layers in the order a request travels, naming the component and what it hands to the next one. Attach a failure to each layer, since that is how the interviewer checks you have run one of these clusters. Close by naming what the integer cannot express, which is the door to every follow-up.

A strong answer

A typical situation: a new platform engineer applies a training manifest, the pod sits in Pending for ten minutes, and kubectl describe pod says "0/12 nodes are available: 12 Insufficient nvidia.com/gpu". Twelve nodes with 96 GPUs between them, and none of them can place a 4-GPU pod. Understanding the path is what turns that message into a diagnosis.

The request is a limit on the container:

resources:
  limits:
    nvidia.com/gpu: 4

Layer one is the device plugin, a DaemonSet that enumerates GPUs through NVML and registers nvidia.com/gpu as an extended resource with the kubelet, which reports it in the node's allocatable. If the plugin is not running, or cannot load NVML because the driver is missing, the node reports zero GPUs. That is the message above: the GPUs exist physically, and Kubernetes has never been told.

Layer two is the kube-scheduler. It filters nodes to those whose allocatable minus already-requested nvidia.com/gpu is at least 4, scores the survivors, and binds the pod. It does not know which four GPUs it is getting, whether they share an NVLink switch, or whether the pod is one of 32 that must start together. The count is the whole model.

Layer three is the kubelet on the chosen node, which calls the plugin's Allocate with the request. The plugin picks device IDs, ideally ones on the same NVLink or PCIe switch if it is configured for that, and returns the device paths and environment to inject.

Layer four is the container runtime with the NVIDIA container toolkit. It bind-mounts the device nodes and the driver's user-space libraries from the host, and sets CUDA_VISIBLE_DEVICES=0,1,2,3 (or the GPU UUIDs). The container's CUDA runtime must be compatible with the host driver; a container built for a newer CUDA than the host driver supports fails here with "CUDA driver version is insufficient", after scheduling succeeded.

The integer model has a cost that shows up as stranded capacity. Work it for a single node:

inputs:  node has 8 GPUs; incoming pods request 3 GPUs each
         the scheduler places a pod only if free ≥ requested
placement: pod 1 takes 3 (5 free), pod 2 takes 3 (2 free), pod 3 needs 3 > 2 → cannot place
stranded = 8 − 6 = 2 GPUs, 25% of the node, idle while a 3-GPU pod waits
across 12 such nodes: 24 GPUs stranded, and a 4-GPU pod finds no node with 4 free
sanity: the cluster reports 96 allocatable and 72 requested, so the dashboard reads 75% busy
        while the Pending pod's message says no node has room; both are true

That is fragmentation, and the scheduler cannot fix it because it cannot see it as a problem. Kubernetes GPU Scheduling covers what the integer cannot express: a specific model (fixed by node feature discovery labels and affinity), a fraction (MIG slices or time-slicing), GPUs in one NVLink domain (plugin allocation policy within a node, a topology-aware scheduler across nodes), all-or-nothing placement for a multi-pod job (gang scheduling), and fair sharing between teams (a batch scheduler such as Kueue).

Dynamic Resource Allocation, GA in the 1.34 line, replaces the count with a ResourceClaim. The driver publishes each device with attributes in a ResourceSlice (product, memory, MIG profile, NVLink domain), the claim selects on those attributes, and the scheduler matches claims to slices. Sharing one claim between pods and carving MIG on demand become first-class. As of 2026 both paths coexist: the count is everywhere and every Helm chart understands it; DRA is where mixed fleets and sharing are going.

The decision for a new cluster: start with the GPU Operator's device plugin path because it is what every chart expects, add NFD labels so a job can pin a GPU model, and move to DRA when the first requirement the integer cannot express arrives, which is usually sharing or topology.

The reversal condition: a fleet that needs fractional or attribute-based allocation, where the device plugin's integer count cannot express the request at all and Dynamic Resource Allocation is the only path. Kubernetes GPU Scheduling covers both models, and kubectl describe node showing nvidia.com/gpu: 8 is the check that the plugin layer is healthy. Node Lifecycle: Drain, Upgrade and Return is what happens to that node afterwards.

What interviewers probe next

• "The pod is Running but nvidia-smi inside it shows nothing. Which layer?" The toolkit: the runtime class or the container toolkit hook did not inject the devices, or the driver libraries did not mount; scheduling was fine.
• "And for a 32-GPU training job?" Kubernetes places pods one at a time, so half the job can start and hold GPUs at a collective barrier forever; that needs gang scheduling in front of the scheduler.
• "Two jobs each want half a GPU. What do you do?" The integer cannot say half; expose MIG slices as their own resource, or time-slice, or use a DRA claim with sharing, and say which isolation each gives.
• "Why limits and not requests?" Extended resources must have requests equal to limits; you cannot overcommit a GPU count, so the limit is the request.

Common mistakes

• Saying "Kubernetes schedules GPUs like CPUs." It counts them like CPUs and cannot see inside them, and GPUs are not divisible or overcommittable the way CPU millicores are.
• Reading "Insufficient nvidia.com/gpu" as a capacity problem when the node reports zero allocatable; the plugin is down, and adding nodes changes nothing.
• Blaming the scheduler for a "no CUDA-capable device" error inside the container, which happens two layers later.
• Forgetting the driver-version rule: the container's CUDA must be supported by the host driver, and the pod does not find out until it runs.

Key takeaways

• Four layers: device plugin advertises, scheduler subtracts, kubelet allocates, toolkit mounts. Each has one failure message.
• The scheduler sees an integer; three 3-GPU pods on an 8-GPU node strand 2 GPUs and a 4-GPU pod waits with the cluster reading 75% busy.
• The integer cannot express model, fraction, topology, all-or-nothing or fairness; NFD, MIG, a topology-aware scheduler, gang scheduling and Kueue each fix one.
• DRA replaces the count with an attribute-based claim; both paths coexist in 2026.