expertNVIDIACoreWeaveCrusoe

One node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.

Compare the same benchmark on a healthy node, then follow the evidence through NCCL transport choices, device topology and fabric health. Half the measured bandwidth is a symptom, not a unique diagnosis.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceInterpret NCCL algbw and busbw, then isolate a slow collective with controlled tests. Distinguish rank delays, GPU-memory paths and shared-fabric limits without guessing from bandwidth ratios.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeNVLink, NVSwitch and PCIe provide different GPU communication paths. Learn how their topology and bandwidth affect the placement of parallel work.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.

Foundational

🩺 Fleet Reliability & Observability

GPU Failure Modes and XID ErrorsXID messages report GPU errors that need interpretation alongside logs and workload behavior. Use the code, recovery guidance and recurrence pattern to choose the next check.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHalf the weights are zero. Why did serving not get faster?Closest to this topicWhat does nvidia-fabricmanager do, and what exactly breaks when it is not running?Closest to this topicA 64-node job's all-reduce runs at a third of the expected bandwidth. Walk me through finding the cause.

FEDITOR'S NOTE

Look for a reproducible comparison, correct interpretation of algbw and busbw, and evidence collected before disruptive recovery. A log label or a topology entry alone should not become a hardware diagnosis.

DISCUSSION · 0

No comments yet — be the first to share your approach.