Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.
One node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.
Half the expected bus bandwidth has four plausible causes needing different fixes: a link trained at reduced width, a fabric that never came up, a topology the job did not expect, or a test that measured latency. The sequence of commands that tells them apart, and what each output looks like on a healthy node.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Scored on a diagnostic order that goes from cheapest check to most invasive, correct expectations for each command's output on a healthy HGX node, and the ability to distinguish a hardware fault from a software mapping error from a measurement artifact.
No comments yet — be the first to share your approach.
