Here is a Kubernetes GPU node. Pods requesting nvidia.com/gpu stay Pending. Fix it in front of me and narrate what you check.

TL;DR: Walk the stack from the bottom: does the kernel see the driver (nvidia-smi on the host), does the runtime inject it (nvidia-container-cli info, the containerd runtime class), does the device plugin run and register (its pod log, then kubectl describe node showing nvidia.com/gpu: 8 under Allocatable), and is the node schedulable (taints, cordon, the nvidia.com/gpu toleration on the pod). The common faults are a driver and toolkit version mismatch after an unattended update, a device plugin in CrashLoopBackOff because the runtime class is missing, and a taint the GPU operator placed during a failed validation and never removed.

How to approach it

Say out loud that Pending has two causes that look identical: the node advertises zero GPUs, or the node advertises eight and the pod is not allowed on it. kubectl describe node splits them in one command, so run it first. Then commit to bottom-up: kernel driver, container runtime, device plugin, scheduler. Restarting the device plugin before reading its log is the move that costs the round, because a plugin that cannot find the driver will crash again in twenty seconds and you have learned nothing.

A strong answer

A typical situation: a node was rebooted overnight after a kernel update, came back Ready, and every GPU pod scheduled to it since has sat Pending with 0/40 nodes are available: 1 Insufficient nvidia.com/gpu. The GPUs are fine. Something between the silicon and the scheduler is not.

First the split, because it decides the whole path:

kubectl describe node gpu-17 | grep -A2 -E "Taints|Allocatable" 
  case A: Allocatable nvidia.com/gpu: 0   (or the line is absent)  → the node is not advertising
  case B: Allocatable nvidia.com/gpu: 8, Taints: nvidia.com/gpu=present:NoSchedule
          and the pod lacks the toleration                          → the node is advertising, the pod is refused
kubectl get events -n gpu-operator --field-selector involvedObject.name=gpu-17 shows validator failures in case A

Case A, bottom-up. Layer 1, the kernel driver. On the host (or a privileged debug pod with nsenter): nvidia-smi. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, the module is not loaded for the new kernel. dmesg | grep -i nvidia shows whether the module failed to build or was never built; dkms status on a DKMS install shows it unbuilt for the new kernel. This is the overnight-reboot fault: the kernel moved and the driver did not follow. The fix is a driver rebuild or, on an operator-managed node, letting the driver DaemonSet pod restart after it detects the kernel version.

Layer 2, the container runtime. The NVIDIA container toolkit injects the host's libcuda.so and device nodes into containers. Check nvidia-container-cli info on the host and that containerd's config has the nvidia runtime and that a RuntimeClass named nvidia exists in the cluster. A toolkit built for driver 550 running against a host now on 535 (or the reverse) fails at container start with a library load error inside every GPU pod, including the device plugin's own pod. Containers, Images and GPU Cold Starts covers the compatibility rule; in the room, the sentence is "the driver on the host and the toolkit's expectations have to agree, and an unattended upgrade of one of them is the usual way they stop agreeing."

Layer 3, the device plugin. kubectl -n gpu-operator get pods -o wide | grep gpu-17 shows whether the plugin pod is Running or in CrashLoopBackOff. Its log is the single most informative line in the whole exercise:

kubectl -n gpu-operator logs nvidia-device-plugin-daemonset-xxxxx
  "Failed to initialize NVML: could not load NVML library"   → layer 1 or 2 broke; fix there, not here
  "Detected NVML platform: found NVML library" then
  "Registered device plugin for 'nvidia.com/gpu' with Kubelet" → plugin is fine; look at kubelet or taints
  no plugin pod on this node at all → the DaemonSet's nodeSelector or the operator's label is missing

The plugin registers over a Unix socket in /var/lib/kubelet/device-plugins/; if the kubelet restarted after the plugin registered, the plugin must re-register, and an old plugin version that does not watch for that leaves Allocatable at zero until someone restarts it. Kubernetes GPU Scheduling covers the advertise-and-match mechanism; the debugging fact is that Allocatable comes only from that registration.

Case B, the pod is refused. The GPU operator taints nodes while its validator runs and removes the taint on success; a validator that failed (often because layer 1 or 2 was broken at the time) leaves the taint behind after the underlying fault heals. kubectl describe node shows it; kubectl taint nodes gpu-17 nvidia.com/gpu:NoSchedule- removes it, and the right fix is to re-run the validator so it removes its own taint. A cordoned node from a maintenance script that never uncordoned is the other version, visible as SchedulingDisabled in kubectl get nodes.

What the outage costs, so the urgency is a number:

node: 8 × H100 at $2.50 per GPU-hour = $20 per hour idle
found at 09:00 after an overnight reboot at 02:00: 7 h × $20 = $140 of idle capacity
plus the queue: a 64-GPU gang that needed this node as its eighth waited the same 7 h
sanity: the node cost is small; the gang delay is 64 × 7 = 448 GPU-hours of other jobs
        held back by one node, which is why node health gates matter more than node cost

The condition that changes the order: if kubectl describe node shows 8 allocatable, skip layers 1 to 3 entirely and go to taints, tolerations and resource requests on the pod (a pod requesting nvidia.com/gpu: 8 on a node with 6 free is also "Insufficient").

The reversal condition: kubectl describe node already showing 8 allocatable GPUs. Then the driver, toolkit and device plugin are all healthy, the fault is above them in scheduling or admission, and every layer below is a wasted check. Kubernetes GPU Scheduling is the layer to open instead.

What interviewers probe next

• "nvidia-smi works on the host but the plugin logs NVML failure. What is between them?" The container toolkit: the plugin runs in a container and needs the runtime to inject the library; check the RuntimeClass and containerd's default runtime.
• "How do you keep this from recurring?" Pin the kernel and driver together (no unattended kernel updates on GPU nodes), and put a validator job in the node's readiness gate so a node with zero advertised GPUs never reads Ready.
• "The node advertises 8, the pod tolerates the taint, still Pending." Read the scheduler's event on the pod: a resource other than the GPU (hugepages, RDMA device, memory) is short, or an affinity rule excludes the node.
• "Would you drain and reimage instead?" On a fleet of hundreds with automation, yes, after capturing the logs; in an exercise, the interviewer wants the diagnosis, and reimaging skips it.

Common mistakes

• Restarting the device plugin pod first; it crashes again and the log you needed is now the new pod's.
• Treating "Insufficient nvidia.com/gpu" as a plugin problem when the node has a taint and eight allocatable GPUs.
• Forgetting that the device plugin itself is a GPU container and fails from the same toolkit mismatch that breaks user pods.
• Declaring the node fixed when nvidia-smi works, without confirming Allocatable went back to 8.

Key takeaways

• kubectl describe node first: zero allocatable means a stack fault; eight allocatable plus Pending means a taint, toleration or request problem.
• Bottom-up: driver (nvidia-smi, dmesg), toolkit (nvidia-container-cli info, RuntimeClass), device plugin (its log, the registration line), scheduler (taints, cordon).
• "Failed to initialize NVML" in the plugin log means fix a lower layer, not the plugin.
• A node idle for 7 hours costs $140 and holds back every gang that needed it.