TL;DR: In the device-plugin path, the plugin reports devices to the kubelet, the node exposes an integer resource, and the scheduler finds a node that satisfies the pod's requests and placement rules. After binding, the kubelet selects device IDs and calls the plugin's allocation interface; the runtime makes those devices accessible. Scheduling a count and preparing working GPU access are separate steps.
How to approach it
Trace the request through each owner. Separate total allocatable capacity from requests already reserved, and separate node selection from device selection. Finish by proving that a container can perform useful GPU work.
A strong answer
A pod is the unit Kubernetes schedules, one or more containers that run together on one node; the kubelet is the agent on each node that starts them; Pending means no node has been chosen yet. A four-GPU pod stays Pending even though a dashboard shows 24 GPUs free. That can be perfectly consistent: if the free devices are spread across twelve nodes, two per node, no single node can host the pod. Kubernetes GPU Scheduling introduces the resource model; the missing detail is where the count becomes physical devices.
The Kubernetes device-plugin interface includes registration, device health reporting and allocation. The plugin's list of healthy devices contributes to node capacity and allocatable resources. An unhealthy device can reduce allocatable capacity; a device already requested by a pod remains part of allocatable capacity but is unavailable to another exclusive allocation.
For a conventional exclusive GPU resource, the request is an integer. The GPU scheduling documentation describes declaring GPU limits and the corresponding request behavior. A request of four does not promise four times the speed, and an unused allocated GPU is not automatically available to another pod.
One node: 8 allocatable GPUs
Existing pod requests: 3 + 3 = 6
Available for another exclusive request: 8 − 6 = 2
Twelve such nodes: 12 × 2 = 24 free GPUs
A single pod requesting 4: zero eligible nodes on GPU capacity alone
Sanity: cluster-wide free capacity is not a per-node placement.
The scheduler also checks CPU, memory, affinity, taints, scheduling state and other applicable constraints. Gang Scheduling with Kueue and Volcano covers coordinated admission. A multi-pod job can span nodes, but its workers need coordinated admission and startup if they depend on each other. One pod does not split its containers across nodes to collect four GPUs.
Once the scheduler chooses a node, the kubelet's device manager chooses the GPU IDs. The plugin may help through its preferred-allocation interface. The kubelet then calls Allocate, and the plugin returns the information needed to expose those devices to the container. Depending on the integration, this can include device nodes, mounts, environment settings or CDI references.
The container's device numbering need not match the host's numbering. A request for four GPUs therefore does not promise host devices 0 through 3. When debugging, record stable device UUIDs so you can follow the same physical GPU across both views.
A node advertising eight is not proof that every future allocation will succeed. Stale health information, runtime setup and application compatibility can still break startup. Read pod events to distinguish a scheduling rejection from a bound pod failing during container creation. Validate the requested device count inside a representative container and run a small correctness workload before accepting a repaired node.
For smaller jobs, change the resource contract deliberately. MIG resources represent supported partitions; configured time-slicing advertises shared access with different isolation properties. Dynamic Resource Allocation is another API path with version- and driver-specific capabilities. None of these means writing 0.5 against an ordinary integer extended resource.
Prefer exclusive whole GPUs when the workload needs them or sharing fails its isolation and latency requirements. Reconsider that choice when measured small workloads fit a supported partition or sharing configuration with acceptable interference. Preserve capacity for larger jobs; packing every node with small requests can create the 24-free-but-unplaceable case above.
What interviewers probe next
- "Does Allocatable fall when a pod reserves four?" The scheduler accounts for requests separately; allocatable is not the remaining-free counter.
- "Who chooses the physical IDs?" The kubelet device manager, with any supported plugin guidance, before allocation preparation.
- "Can low GPU utilization trigger sharing automatically?" Not in the ordinary exclusive request contract.
Common mistakes
- Adding free GPUs across nodes to satisfy one pod.
- Treating a bound pod as proof that CUDA startup succeeded.
- Assuming visible ordinal zero identifies the same physical GPU everywhere.
Key takeaways
- 8 allocatable − 6 requested = 2 available.
- Twelve nodes with two free GPUs each cannot fit one four-GPU pod.
- Trace node selection, device allocation and useful execution separately.