device plugin

The four layers between `nvidia.com/gpu: 4` and a container that can run CUDA, the one failure each layer produces, the stranded-GPU arithmetic the integer model causes, and why Dynamic Resource Allocation replaces the count with a claim.

A node with eight healthy H100s that Kubernetes thinks has none. The layered check from kernel driver to container toolkit to device plugin to taints and allocatable, in the order that isolates the fault fastest, with the log lines each layer prints when it is the one that broke.

A GPU node needs a driver, a container toolkit, a device plugin, a metrics exporter and a validator, in that order, and each needs to know which nodes it belongs on. The operator's component chain, the labels NFD writes to trigger it, and the bring-up arithmetic that rules out doing it by hand.

Device plugins count GPUs and know nothing else; Dynamic Resource Allocation lets a pod ask for devices by attribute and share them by claim. What the two models can and cannot express, what DRA changes for MIG, NVLink domains and multi-node gangs, and a migration stance for a fleet that runs both.

A pod's default network is an overlay that no RDMA NIC will speak. The three ways to give a pod the fabric (host networking, SR-IOV virtual functions via a NIC device plugin, or a macvlan secondary interface), what each costs in isolation and speed, how NCCL finds the right NIC, and the failure that halves bandwidth.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.