TL;DR: First determine whether the pod has a node assignment. An unscheduled pod needs scheduler-event and resource analysis; a bound pod failing to start needs allocation/runtime investigation. Then follow evidence through device discovery, driver and fabric health. An allocatable count or successful device listing alone does not certify a working GPU workload.
How to approach it
Preserve events and logs before restarting components. Identify the failing boundary, compare one affected node with a known-good peer, and change the smallest justified component. Verify both Kubernetes resource reporting and real GPU execution before returning the node to service.
A strong answer
A GPU pod is Pending after a node update. That status covers more than one stage: it may be waiting for a scheduler assignment or for container setup on an assigned node. Start with the pod's conditions, node assignment and recent events. Kubernetes GPU Scheduling explains the allocation path; here the goal is to locate where this request stopped.
For an unassigned pod, inspect its GPU resource name and quantity, CPU/memory requests, affinity (rules about which nodes the pod requires or prefers), tolerations (permission to run on nodes tainted, or marked, for special use) and any workload admission conditions. Compare these with candidate nodes, their scheduling state and existing reservations. A scheduler message about insufficient GPUs does not prove that the plugin failed.
Node allocatable GPUs: 8
Already requested by other pods: 6
New request: 4
Available = 8 − 6 = 2; shortfall = 4 − 2 = 2
Sanity: an allocatable count of eight is compatible with an insufficient-GPU event.
If the advertised resource is absent or unexpectedly low, inspect the device-plugin pod on that node, its health reports and kubelet registration. A missing pod may reflect a selector, toleration or installation issue. A running plugin may mark individual devices unhealthy. Capacity and allocatable can differ for legitimate health reasons; compare stable device identities rather than guessing from the total alone.
If the pod has a node but container creation fails, read the precise runtime event. A missing runtime handler, image failure, mount failure and device-allocation failure need different fixes. RuntimeClass is relevant when that integration uses it; do not assume every CDI or runtime configuration requires the same handler name.
NVIDIA's troubleshooting guide describes examples including operand initialization failures, missing runtime handlers and unhealthy devices. Use the section matching the observed failure and installed version. A generic claim that toolkit version X must match driver version X is not a compatibility rule; check the supported stack and the actual library or module error.
If the evidence points to the driver, check whether the host can see the devices and read the kernel and module logs. Follow the installation method used on that node rather than replacing components independently. A device listing proves discovery, but compute or fabric initialization can still fail. Include the fabric service on relevant NVSwitch systems. If libraries work on the host but are missing in the container, follow how the runtime exposes them and check permissions. Save the logs before resetting or reimaging the node.
Taints and cordons have owners. Find the controller or maintenance process that set them and determine whether its condition is resolved. Removing a taint may make a pod schedulable while bypassing the reason the node was excluded. Do not assume every GPU Operator validator manages a universal readiness taint; use the actual deployment's health gate.
The repair needs two acceptance checks: the intended resource is advertised and available as expected, and a representative container receives the intended device and passes a correctness workload. Then test the workload class that failed, including collective communication where applicable. Monitor for recurrence after returning capacity.
A synthetic accounting example keeps impact honest: eight idle GPUs for seven hours are 56 idle GPU-hours. At an assumed $2.50 per GPU-hour, that is $140 of capacity cost. A 64-GPU job delayed seven hours has 448 requested GPU-hours of waiting exposure, but those other GPUs are not necessarily idle; they may serve other jobs. Keep queue delay and wasted capacity separate.
Reverse the initial stack diagnosis if the scheduler evidence shows that all requested capacity is already allocated or a placement rule excludes the node. Conversely, a successful scheduling decision does not eliminate runtime or hardware faults. Follow the next failed boundary.
What interviewers probe next
- "The host lists eight GPUs. Is it fixed?" No; test container access and useful compute.
- "Should I restart the plugin?" Only after preserving evidence and identifying a reason it will recover.
- "What if the count is stale?" Compare current plugin/kubelet observations and actual allocation behavior.
Common mistakes
- Treating every Pending pod as unscheduled.
- Removing maintenance taints without resolving their condition.
- Reporting all delayed job capacity as idle capacity.
Key takeaways
- Eight allocatable minus six requested leaves two, so a four-GPU request can fail normally.
- Separate assignment, container creation and GPU execution.
- Verify the repaired workload path, not only the device listing.