TL;DR: XID 79 means the driver found the GPU inaccessible over PCIe. Follow the applicable host-restart recovery guidance, preserve evidence and stop unsafe allocation. PCIe path, device, power and software issues can be involved. A successful reboot restores availability; it does not establish the cause or prove a repeat event harmless.
How to approach it
Identify the device and affected workloads, preserve preceding logs, and apply the supported recovery scope. Then run acceptance checks and review recurrence across device, slot and software history. Keep recovery attempts bounded so a failing node cannot cycle endlessly through production.
A strong answer
PCIe is the expansion bus that connects a GPU to its host CPU, so losing PCIe access means the operating system can no longer talk to the device at all. Suppose a training job loses access to one GPU. Rebooting brings it back, but the same failure occurs during the next shift. The reboot solved the immediate availability problem while leaving the cause unresolved. GPU Failure Modes and XID Errors explains the code; Incident Response for GPU Fleets covers who owns the recovery and when to escalate.
The NVIDIA XID catalog identifies XID 79 as GPU inaccessibility over PCIe, with bare-metal restart as the immediate recovery bucket and support investigation for recurrence. It notes PCIe-link hardware failures, GPU hardware and other driver issues as possible causes. Do not infer that the device must be absent from every enumeration interface, or that software can never be part of the cause.
Record the GPU’s stable identifier (UUID), its PCI address, the host and the affected job attempt. Save timestamps, driver and firmware versions, kernel PCIe errors, and any nearby power or temperature events. Another worker may time out because this GPU disappeared, so check the order of events before opening a second investigation. Stop assigning new work to the affected device or host as the platform requires; that step should not depend on the failed application replying.
Coordinate the affected workload's recovery. A rigid multi-node job may need a restart from its last trusted durable checkpoint. It may be unable to create a new valid checkpoint after losing a rank. Independent workloads on the host need their own disruption handling before host recovery. A device reset is not a universal substitute for the documented restart path; follow the supported procedure and escalate if it fails, including power-cycle requirements where applicable.
After recovery, verify device enumeration, supported diagnostics and representative compute and communication. Use agreed baseline ranges, not a promise of nominal rated bandwidth. A device can pass a short test and fail under another workload or operating condition. Keep repeat history attached to stable physical identities even when the node is reimaged.
For recurrent faults, compare candidate causes with evidence. A failure moving with a card supports a device-related explanation; staying with a slot supports the slot or path. These are controlled maintenance tests to plan with qualified support, not conclusive results from one swap. Shared firmware, power and environmental conditions can follow several parts of the experiment. Do not rank causes by invented percentages.
A statistical example explains why a repeat count should trigger investigation rather than certainty. Assume, only for this model, a stationary Poisson incident process with mean 0.1 events per node per month:
P(at least 3 in a month) = 1 − exp(−0.1) × (1 + 0.1 + 0.1²/2)
≈ 0.00015465 per node-month
Across 5,000 node-months: expected such cases ≈ 0.773
Sanity: a rare result for one preselected node can appear in a large fleet.
The model does not prove that any observed cluster is random, and real faults may be correlated. It shows why a threshold is an operational decision, not a mathematical hardware diagnosis. Use a local XID-79 incident baseline, not an all-cause training-interruption rate normalized by GPU-hours.
Set a bounded recovery policy with explicit quarantine and support escalation conditions. It can become stricter for repeated unexplained faults, large blast radius or failed diagnostics, without pretending one fixed count applies everywhere. Measure time to isolate, successful recovery, recurrence and useful job loss.
Keep the node out of service if acceptance tests fail or repeated faults remain unexplained. If controlled tests instead point to the host’s PCIe path or a supported software fix, replacing the card may not help. Running with seven responsive GPUs is an option only when the system and scheduler support that configuration.
What interviewers probe next
- "Does a reboot identify the cause?" No; it only demonstrates recovery under that attempt.
- "Can the job checkpoint after the error?" It may not have intact state or all required ranks; use the trusted durable checkpoint.
- "Why not loop reboots?" Repetition can hide evidence and keep exposing jobs to unresolved faults.
Common mistakes
- Declaring every XID 79 a proven bad GPU.
- Treating a passing short diagnostic as a permanent health certificate.
- Using a single-node tail probability as conclusive fleet-wide evidence.
Key takeaways
- XID 79 reports PCIe inaccessibility, not a unique cause.
- In the stated Poisson model, about 0.773 three-event cases are expected across 5,000 node-months.
- Preserve evidence, bound recovery and keep recurrence history.