TL;DR: Repeat the slow-rail result with the same workload and an unaffected rail as a control. Verify GPU/NIC mapping and routing, then use a pairwise test matrix to narrow the affected endpoints or shared paths. Read negotiated link settings and counter deltas while the test runs. A 15% gap is evidence of an asymmetry; it does not by itself identify a cable fault, and a fleet-wide deficit can still have a physical cause.
How to approach it
Define the benchmark and expected result first. Control placement and traffic, localize the deficit, then test a specific cause. Finish by rerunning both the isolated test and the representative collective after the repair.
A strong answer
A rail is the set of network paths used by the same GPU position in every node, so rail 3 carries the traffic of GPU 3 across the cluster. Suppose a 512-GPU acceptance report shows seven rails near the reference bandwidth and rail 3 about 15% lower. In a rail-optimized fabric, corresponding GPU/NIC positions use corresponding network paths. That gives a useful comparison, provided hardware, routing, traffic and test configuration really match. Rail-Optimized and Fat-Tree Fabrics explains the arrangement; symmetry is an expectation to verify, not a guarantee of identical traffic.
Be precise about the reported bandwidth
For eight independent equal-capacity rail tests, a sum can hide one weak result:
Healthy rail result: B
Seven healthy rails plus one at 0.85B: 7B + 0.85B = 7.85B
Expected sum: 8B
Deficit in the sum: (8B - 7.85B) / 8B = 1.875%
That calculation is not a prediction of all-reduce slowdown. A collective may wait on a critical path, rebalance traffic or overlap communication with compute. Its effect depends on message sizes and algorithm. The NCCL tests performance guide also distinguishes algorithm bandwidth from its normalized bus-bandwidth metric; neither is simply a sum of physical port rates.
Find the smallest repeatable case
Repeat the affected and control tests with the same message sizes, direction, duration, warmup, GPU/NIC binding and CPU affinity. Record variation and competing traffic. Inspect the selected transport and device mapping rather than assuming the test used the intended rail.
Then compare endpoint pairs within the slow rail. Begin with pairs that stay within a leaf and pairs that cross the relevant uplinks, using known-good endpoints as controls. A complete all-pairs test may be expensive; choose a matrix that distinguishes shared paths before expanding it.
| Pattern | Hypotheses to test |
|---|---|
| Many paths sharing an uplink are slow | Uplink capacity, routing, congestion, configuration or physical health |
| Most slow pairs include one host | NIC/GPU mapping, PCIe path, NUMA placement, host load or that host's link |
| Only large messages fall behind | Sustained transfer limits, congestion, retransmission or protocol behavior |
| Gap disappears when tested alone | Contention or scheduling effects; still inspect what caused them |
These patterns overlap. Several bad links can imitate a shared switch problem, and a misconfigured test can imitate a bad host. Debugging a Slow All-Reduce develops this comparison method.
Rail 3 is repeatedly 15% below a matched control.
- Hold the test configuration fixed
Confirm NIC selection, GPU binding, traffic, message size and direction.
- Compare paths that share components
Contrast same-leaf and cross-leaf pairs, then isolate hosts and uplinks.
- Correlate the test with counters
Check negotiated modes and error/congestion deltas; change one suspected cause and repeat.
Check both fabric and host paths
On InfiniBand, use the supported port and fabric tools to compare negotiated width/rate with the intended topology. On Ethernet, inspect the corresponding switch and NIC diagnostics. Record counter deltas over the test interval, including relevant physical errors, recoveries, drops, congestion and pause/flow-control signals where supported. Counter names and meanings vary by generation and protocol.
A reduced link rate is a finding, not the end of the investigation. Determine whether configuration, an unsupported assembly, a damaged lane or another issue caused it. Full rate with rising errors supports investigating signal quality, but it does not uniquely identify the cable. Preserve the evidence before a controlled swap or configuration change. Cables, Transceivers and the Optics Power Budget covers assembly compatibility.
Inspect the NIC's actual PCIe link and upstream path as well as its advertised capability:
PCIe Gen5 x16 encoding-limited rate, one direction
32 GT/s × 16 × (128/130) / 8 ≈ 63.0 GB/s
Gen5 x8 ≈ 31.5 GB/s
800 Gb/s network line rate / 8 = 100 GB/s
sanity: payload rates are lower, and the narrowest active path can limit a transfer.
Do not assume an 800G NIC always has a single Gen5 x16 connection. ConnectX-8 supports Gen6 x16, and the installed SKU and topology determine the usable host path. Conversely, if the workload was already limited elsewhere, halving a link's width need not halve measured application throughput. A 15% deficit cannot rule PCIe out just by its size.
Repair and verify the same workload
Once a controlled test identifies a cause, correct it and rerun the original pairwise matrix, per-rail tests and full collective. Save configuration and counter baselines with the acceptance report. Use an absolute performance floor alongside peer comparison: eight equally slow rails can pass a relative test while all missing the requirement.
A sample rule such as “within 3% of matched peers” is only useful if repeatability is substantially tighter than 3% and the reference itself meets the target. Set tolerances from the test and hardware variation. If all rails are low, shift attention toward common configuration, expectations and shared paths, while retaining common physical limitations as candidates.
What interviewers probe next
- “Why not replace the cable immediately?” A documented controlled swap is useful once the baseline and compatibility are known. An untracked swap is weak evidence.
- “How many tests will isolate it?” There is no guaranteed four-step bisection; path overlap and multiple faults determine the required comparisons.
- “Does the collective lose exactly 15%?” No. Measure its critical path and application effect separately from the isolated rail deficit.
Common mistakes
- Applying the 1.875% sum calculation directly to collective completion time.
- Assuming a full-rate link is healthy or a reduced-rate link explains its own cause.
- Reading lifetime counters without matching the interval to the test.
- Excluding physical faults because every rail is affected.
Key takeaways
- Per-rail results expose a deficit that an independent-bandwidth sum can dilute to 1.875%.
- Use matched tests and path-aware comparisons to choose the next measurement.
- Verify both the repaired path and the real collective before accepting the system.