Advanced

NVLink and Fabric Faults

GPU link faults can slow collectives or stop distributed progress. Correlate application timing with link counters and controlled path tests to locate the problem.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

NVLink, NVSwitch and PCIe GPU Failure Modes and XID Errors DCGM and GPU Telemetry Stragglers and Hangs Debugging a Slow All-Reduce RDMA, InfiniBand and RoCEv2

LESSONS THAT TEACH THIS

Training at ScaleFind the late dependency before blaming the rank that waits22 min→Fleet and ReliabilityClassify fault evidence, affected work and recovery action15 min→

PRACTICE THIS IN REAL QUESTIONS

GPU Fleet Reliability & ObservabilityWhat do NVLink errors look like in telemetry, when is a link degrading rather than broken, and when do you drain the node?→Hardware, Cabling & Cluster Build-OutWhat does nvidia-fabricmanager do, and what exactly breaks when it is not running?→GPU & Accelerator ArchitectureOne node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.→Hardware, Cabling & Cluster Build-OutNew nodes arrive with no software. Walk me through bring-up, and say why the order matters.→GPU Fleet Reliability & ObservabilityWhat is an XID error, which ones mean the hardware is bad, and which ones mean somebody's kernel has a bug?→GPU Fleet Reliability & ObservabilityHow do GPUs actually fail at fleet scale, how often, and which failures should the platform expect to handle every day?→