GPU Fleet Reliability & Observability: the practice test

XID 79 means the host lost contact with the GPU over PCIe: nvidia-smi hangs or shows the device missing and every process using it is stuck. Nothing short of a reboot brings it back. The reflex is to drain the node, restart the job on a spare from its last checkpoint, then reboot and diagnose (power, thermal, board). A GPU that falls off the bus twice within a window goes to RMA rather than back into the pool; the recurrence rule is what keeps a bad board from stopping the next job.

The utilization field is time-with-any-kernel-active over the sample window. A decode loop at batch 1, a chain of small elementwise kernels, or a memory-bound attention kernel all report 100% while using a sliver of the chip. The fields that say what the GPU is doing are the DCGM profiling counters: SM active, SM occupancy, tensor pipe active (which tracks MFU) and DRAM active. A fleet alert on utilization dropping below 90% fires on every loader stall and misses every thermal throttle.