During acceptance, seven rails hit expected bandwidth and one is 15 percent low. Find it.

TL;DR: Symmetry is the tool. In a rail-optimized cluster the eight rails carry identical traffic across identical hardware, so a 15 percent gap on one rail is a defect and not a tuning question, and it is localizable by bisection. Run pairwise bandwidth tests within the slow rail to find whether the deficit is uniform across all node pairs, which points at the switch or its configuration, or concentrated in pairs involving particular nodes, which points at those nodes. Then read per-port counters on the implicated links. Three physical causes account for almost all of these: a link running at a reduced width or speed because it negotiated down, a marginal link accumulating symbol errors and retransmitting, or a node whose NIC sits behind a narrower PCIe path than its siblings. Each has a distinct signature in ibdiagnet and the per-port counters, so the diagnosis is a lookup once the location is known. Do not average across rails, because averaging is what hides this.

How to approach it

Confirm the asymmetry is real and repeatable before investigating, because a single measurement on a busy fabric is noise. Then bisect: rail, then switch, then node, then port. Read counters at each level rather than swapping parts. Then classify the physical cause. Close with the acceptance criterion that would have caught it automatically, since this fault is exactly what per-rail acceptance testing exists for.

A strong answer

A typical situation: acceptance testing on a 512-GPU scalable unit reports aggregate all-reduce bandwidth about 2 percent below expectation. Averaged over the whole unit that looks like noise. Broken out per rail, seven rails are at the expected number and rail 3 is 15 percent low, and the aggregate hides it because one rail of eight contributes an eighth of the total.

Why the aggregate hid it, which is the first lesson:

the arithmetic of hiding
  eight rails, one of them 15% low
  aggregate deficit = 15% / 8 = 1.875%
  measured aggregate deficit: about 2%
sanity: a 2% aggregate miss and a 15% single-rail fault are the same observation, and only
        the per-rail breakdown distinguishes "close enough" from "one rail is broken", which
        is why acceptance criteria are written per rail

The bisection, four steps from 512 links to one:

step 1: is it the rail or the test?
  rerun rail 3 alone, three times, and rerun rail 4 alone as a control
  repeatable gap -> real; varying gap -> contention or a noisy neighbour, investigate that

step 2: uniform across the rail, or concentrated?
  pairwise bandwidth between node pairs within rail 3, using ib_write_bw or nccl-tests
    restricted to that rail
  every pair low by a similar amount     -> the leaf switch, its configuration, or its
                                            uplinks to the spine
  only pairs involving nodes 17 and 41   -> those nodes
sanity: this single test splits the causes into two disjoint sets and takes minutes, and it
        is the step people skip in favour of reading logs

step 3: which port
  on the implicated side, list the ports and their negotiated rate and width
    ibstat and ibstatus per host
    ibdiagnet across the fabric, which reports links below expected width or speed directly
  a port at a reduced width or speed is the answer and the investigation is over

step 4: if all ports report full rate
  read the error counters over a fixed interval, not cumulatively:
    SymbolErrorCounter        rising means signal integrity
    PortRcvErrors             rising means received errors
    LinkErrorRecoveryCounter  rising means the link is recovering rather than failing
    LinkDownedCounter         rising means it is dropping and retraining
  errors accumulating on one port under load is a marginal link

Debugging a Slow All-Reduce covers the general methodology, and Rail-Optimized and Fat-Tree Fabrics explains why the eight rails are identical by construction, which is the property this whole procedure rests on. Cables, Transceivers and the Optics Power Budget covers the physical layer this lands in.

The three causes, and how the counters tell them apart:

the arithmetic that makes the third cause recognizable
  ConnectX-8 at 800 Gb/s = 100 GB/s each direction
  PCIe Gen5 x16 delivers about 64 GB/s each direction
  PCIe Gen5 x8 delivers about 32 GB/s
  so a NIC that came up at x8 instead of x16 caps at roughly a third of the port's rate
  15% low is not that, so this case would show a much larger deficit
sanity: matching the size of the deficit against what each cause can produce narrows the
        list before any counter is read, and a 15% gap is more consistent with retransmission
        than with a halved link width

The reversal condition: if all eight rails are equally low against the expected number, this is not a physical fault and the bisection wastes time. Equal rails point at something global: a NCCL algorithm or protocol selection, an environment variable that differs from the reference configuration, a firmware version across the fleet, or an expectation derived from the wrong hardware. The way to tell in one step is exactly the per-rail breakdown that started this, which is why it belongs in the acceptance report by default rather than being produced during an investigation.

What interviewers probe next

• "Why not just replace the cable?" Because the counters distinguish three causes with different fixes, and replacing without reading them means the next occurrence starts from nothing.
• "How would you catch this automatically?" A per-rail acceptance criterion with a tolerance, such as every rail within 3 percent of the median, reported before sign-off.
• "What if it appears months later?" Same bisection, plus a change record check. A rail that was fine and is now slow points at a change: firmware, a cable moved during service, or a transceiver degrading.
• "Does 15 percent on one rail matter?" Yes, because a collective finishes when the slowest participant does, so a job spanning all rails runs at the slow rail's pace rather than at the average.

Common mistakes

• Reporting aggregate bandwidth without a per-rail breakdown, which turns a 15 percent fault into a 2 percent rounding error.
• Swapping parts before reading counters, which loses the evidence that would identify the cause.
• Reading cumulative counters rather than the delta over an interval, so old errors look like current ones.
• Ignoring the node-side PCIe path, which is a common cause and is visible in lspci in seconds.
• Concluding a physical fault when all rails are equally low, which is a configuration problem instead.

Key takeaways

• A single slow rail of eight shows up as a 1.875 percent aggregate deficit, so per-rail reporting is the only way to see it.
• Bisect in four steps: rail, uniform-or-concentrated, port rate and width, then error counters over an interval.
• Three causes: a link negotiated down, a marginal link accumulating symbol errors, or a NIC on a narrower PCIe path.
• A NIC at PCIe Gen5 x8 caps near 32 GB/s against 64 at x16, so that fault produces a much larger deficit than 15 percent.
• Write per-rail tolerance into acceptance, because a collective runs at the slowest rail's pace and not at the average.