TL;DR: Validate the expected number, reproduce the collective under matched conditions, and vary scale and placement to narrow the cause. A third of the previous bandwidth does not identify a faulty link or imply a threefold slower training step. Separate logical bandwidth, normalized bus bandwidth, rank arrival and exposed communication before changing hardware or NCCL settings.
How to approach it
First ask what “expected” means: a vendor limit, an acceptance run, or last week's same workload. Fix the message size, rank count, dtype, mapping and timing boundary before comparing. Then contrast isolated collectives with the application and local pairs with the full group, attaching counter changes to each experiment. Close by verifying correctness and application progress after a targeted change.
A strong answer
A 64-node job can show a sharp communication regression while every device dashboard remains green. That is enough to investigate, but not enough to replace a cable. Start by preserving the failing allocation, logs and launch configuration so the next run remains comparable.
Translate the report into an experiment. Assume eight ranks per node, one per GPU: 64 × 8 = 512 ranks, not 64. Save the collective type, payload bytes, datatype, process/GPU mapping, runtime versions, in-place setting, warm-up and iterations. Check correctness results before trusting performance. Debugging a Slow All-Reduce is the free reference for interpreting the columns.
For a synthetic 256 MiB all-reduce, suppose a matched baseline takes 20 ms and the suspect run takes 60 ms:
S = 256 × 1,048,576 = 268,435,456 bytes per rank
algbw = S / elapsed_seconds / 1e9
baseline = 268,435,456 / 0.020 / 1e9 = 13.4217728 GB/s
suspect = 268,435,456 / 0.060 / 1e9 = 4.4739243 GB/s
all-reduce busbw factor = 2 × (512 - 1) / 512 = 1.99609375
baseline busbw ≈ 26.79 GB/s; suspect busbw ≈ 8.93 GB/s
sanity: the factor is unchanged. Three times the duration gives one-third the rate.
NVIDIA's performance definitions define this normalization. It does not measure bytes on a particular NIC or establish which collective algorithm ran. A nominal 400 Gb/s port is 50 GB/s before overhead; comparing it to a whole collective needs the actual mapping, shared paths and direction of traffic.
Find the exposed delay. If exactly one non-overlapped collective changes from 20 to 60 ms and the rest of a step remains 980 ms, the step grows from 1,000 to 1,040 ms. Throughput becomes 1,000/1,040 ≈ 96.15% of baseline, a 3.85% decrease. Many collectives or lost overlap can cause a larger impact; measure those dependencies rather than extrapolating from one rate.
Use scale as a controlled variable. Run all_reduce_perf with representative sizes using the project's launcher guidance. Compare local GPU pairs and a node, remote pairs, then the full group. Sweep both small and large messages. Preserve software and affinity where possible, record repeats and alternate baseline/suspect runs to reduce time-of-day confounding. These are instructions for a hardware test, not a claim that one was executed for this answer.
| Result | Next discriminating experiment |
|---|---|
| Local pairs slow | Compare device paths, clocks and competing work |
| Some remote pairs slow | Re-test an endpoint with another peer/path |
| Pairs fast, group slow | Restore contention; inspect shared links and algorithm selection |
| Isolated test fast, job slow | Inspect rank arrival and compute/communication overlap |
If each half of the job runs quickly when tested alone, do not immediately clear every node. Splitting the job removed shared traffic and may have changed the collective algorithm. Bring those conditions back in a controlled test to distinguish congestion from an endpoint problem. The expected bandwidth can also change with rank count and message size, even on a well-provisioned fabric.
Inspect the path that actually ran. NCCL_DEBUG=INFO can help identify selected transports, but log wording varies. nvidia-smi topo -m describes device proximity, not achieved bandwidth. Compare host-buffer and GPU-buffer RDMA tests explicitly: a fast host test does not clear the GPU-to-NIC path. NVIDIA's network troubleshooting guide describes port state/rate checks and the relevant point-to-point tests.
Take port error, retry and congestion-counter deltas during the failing interval. A lifetime total may be old; an ECN mark can indicate functioning congestion control rather than a broken switch. Check whether contention follows traffic through a shared uplink or stays with one endpoint when the peer changes.
Do not infer GPUDirect state from “half speed” or a missing nvidia-peermem module. A supported DMA-BUF path is another possibility, documented in NVIDIA's GPU troubleshooting guide. Investigate registration and selected transport evidence for the installed stack. Changes to host isolation or firmware deserve a platform-specific plan, not a blanket tuning command.
Collective Communication Primitives explains why a late rank can make others wait. Align timeline clocks before comparing hosts; inspect local duration measurements when clock alignment is uncertain. Intermittent faults need observations over the incident window, not a clean two-minute sample.
Change the implicated configuration or component, repeat the matched tests, then restore the application's load. Keep an algorithm override only if it is a supported, documented improvement for the required workloads. If small-message latency dominates instead, evaluate fewer/larger collectives and their effect on overlap. A faster isolated collective that delays useful compute may still slow training.
What interviewers probe next
- "Both halves are fast. Are all nodes healthy?" No. The split removed shared traffic and may have changed algorithm selection. Restore those conditions deliberately.
- "Only this job is slow. Can hardware be excluded?" No. Its placement or traffic pattern may exercise a path other jobs avoid. Compare matched mappings and timings.
- "How do you distinguish a cable from a NIC?" Change one approved component or path at a time against a known-good control, preserving mappings and counter evidence. A slow pair alone identifies a path, not one component.
Common mistakes
- Using node count where the normalization requires participating rank count.
- Treating normalized bus bandwidth as a physical port counter.
- Declaring GPUDirect disabled from a ratio or one absent module.
- Losing the failing baseline through several simultaneous tuning changes.
Key takeaways
- With eight ranks per node, 64 nodes means 512 ranks and a 1.99609375 all-reduce factor.
- A 20-to-60 ms collective increase need not triple the full step; identify exposed time.
- Keep the counter interval, tested memory path and application workload attached to the diagnosis.