GPU Failure Modes and XID Errors
XID messages report GPU errors that need interpretation alongside logs and workload behavior. Use the code, recovery guidance and recurrence pattern to choose the next check.
TL;DR: An XID narrows the investigation but rarely proves the root cause by itself. Preserve the message and surrounding evidence, assess affected work, and follow the recovery guidance for that GPU and driver. Drain, reset or replace hardware when the evidence and supported procedure justify it.
What an XID is
The NVIDIA kernel driver (nvidia.ko, logged as NVRM) reports errors it detects in the GPU as XID events: an integer code, the PCI address of the GPU, and some context. They appear in the kernel ring buffer (the operating system's in-memory log of kernel messages, read with dmesg) and in /var/log/syslog or journalctl -k, and DCGM reports them as a field. The code tells you which subsystem detected the problem; the context (a channel id, an address, a link number) narrows it down; and the count and recurrence tell you whether it is an application bug, a transient, or a board that is dying.
[Tue Sep 1 03:12:44 2026] NVRM: Xid (PCI:0000:4e:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[Tue Sep 1 03:12:44 2026] NVRM: GPU 0000:4e:00.0: GPU has fallen off the bus.
That line is the canonical fleet-page: the GPU at PCI 4e stopped responding to the host. nvidia-smi will hang or report the GPU missing, and every process using it is stuck.
The taxonomy
Use the NVIDIA XID catalog for the GPU and driver in service. These groups help organize the investigation; they do not assign blame by themselves.
| Codes | Area to investigate | Useful next evidence |
|---|---|---|
| 13, 31 | execution exception or memory access fault | application trace, sanitizer results, recurrence across devices |
| 43, 45 | stopped channel or cleanup | the earlier error and affected process |
| 48, 63, 64, 92, 94, 95 | ECC, containment or remapping | current recovery action, remap state and diagnostic results |
| 74 | NVLink | link and fabric-manager logs, affected peers |
| 79 | device no longer responding over PCIe | host, power, PCIe and driver evidence |
| 119, 120 | GSP RPC | firmware, driver and recovery guidance |
The same symptom across independent jobs on one device raises suspicion about that device. The same workload failing across devices raises suspicion about the workload. Neither pattern is proof: shared software, configuration and infrastructure can produce both.
The three decisions
The operational choices are to retry affected work, recover the device, or remove it for diagnosis and possible replacement. Preserve the XID and surrounding logs before a reset discards evidence, when the incident allows it.
A contained memory error may allow unaffected work to continue; an uncontained error can require a reset. A pending remap and a failed remap are different states. Read the reported recovery action and the vendor procedure rather than translating every memory XID into an immediate return request.
For a missing GPU or failed fabric path, stop admitting work to the affected scope, restore service on healthy capacity where possible, and investigate power, physical links and software. Replacement follows diagnostic evidence and the supported criteria. An arbitrary “second error means replace” rule can both return working hardware and miss a serious first event.
How often, and why that number matters
The public reference point is Meta's Llama 3 training report: 16,384 H100s over 54 days saw 466 job interruptions, 419 of them unexpected, and about 78% of the unexpected ones were confirmed or suspected hardware. Of those, GPUs and their HBM were the largest share (faulty GPU about 30%, HBM about 17%), with software bugs, network switches and cables, host maintenance, GPU SRAM, and the GPU system processor behind them.
unexpected interruptions: 419 in 54 days
rate = 419 ÷ 54 ≈ 7.8 per day ≈ one every 3.1 hours across the whole fleet
per-GPU MTBF, treating every interruption as one GPU-attributable event (an overestimate of GPU blame):
MTBF_gpu ≈ 16,384 × 54 × 24 GPU-hours ÷ 419 ≈ 50,700 GPU-hours ≈ 5.8 GPU-years between failures
(16,384 × the rounded 3.1 h gives 50,800; the difference is rounding, not a finding)
so on a fleet of N GPUs, the expected time between unexpected stops is
T ≈ 50,700 h ÷ N
N = 1,024: ≈ 50 h (about two days); N = 16,384: ≈ 3.1 h; N = 100,000: ≈ 30 min
sanity: a per-GPU lifetime of several years sounds fine, and at 100k GPUs it becomes a stop every
half hour, which is why checkpoint cadence and automatic restart dominate at scale.
This is the derivation behind every fleet reliability design: failures that are rare per device are continuous per fleet, and the arithmetic gives the interval that checkpointing and restart have to beat (Training Uptime and Interruption Statistics).
Working it in the room
For a GPU that disappears from the bus, explain how you would protect affected work and collect power, thermal, PCIe and driver evidence. Compare recurrence across jobs and devices without treating that pattern as proof. Use vendor recovery and replacement criteria rather than an invented universal number of retries.
What to remember
- An XID records where an error was detected; context is needed to investigate its cause.
- Separate application recovery, device recovery and replacement decisions.
- Preserve logs and inspect containment, remapping and the reported recovery action.
- Recurrence patterns guide investigation without proving hardware or software blame.
- Historical interruption rates illustrate fleet-scale risk; measure the new fleet rather than treating the extrapolation as a forecast.