TL;DR: Save the port states and inventory, then compare failed links with working ones. Check administrative state, speed, lane layout, protocol, FEC and the exact cable or module compatibility at both ends. A length or part-number pattern is a strong lead, not proof. Confirm it with a controlled swap using a supported assembly. There is no universal two-metre limit for every product labelled “800G.”
How to approach it
Establish what changed and which physical links are affected. Separate links that never trained from links that became unstable, while remembering that their causes overlap. Use configuration, inventory and counter evidence to choose a controlled test, then verify the repaired links under load.
A strong answer
FEC (forward error correction) adds redundant bits so a receiver can repair bit errors; both ends of a link must agree on the FEC mode. Suppose 22 of 64 links are down after a row is recabled. First preserve the cable map, switch and host logs, port settings, firmware versions and module inventory. A maintenance window may have changed more than cables. Record both ends of each link so that “port 12” cannot refer to two different devices.
Classify the symptom without ruling out causes too early
| Observation | Useful next checks |
|---|---|
| Port administratively disabled | Intended configuration and change record |
| No physical link ever established | Seating, supported assembly, lane/breakout configuration, speed, protocol and FEC agreement |
| Link establishes, then drops | Error and recovery counter deltas, thermal data, remote resets and firmware logs |
| Link remains up below expected speed or width | Negotiated settings, supported modes and lane health |
A damaged or marginal cable can fail to train or train intermittently. Configuration and firmware faults can also produce several symptoms. Missing error counters do not prove the medium is sound: some counters only become useful after training, and their meaning depends on the device and protocol.
Read the part number, not just the speed label
An aggregate 800 Gb/s connector can represent a different lane arrangement from a single 800 Gb/s port. A twin-port 2×400G assembly is not interchangeable with every 800G implementation. Check connector, signalling generation, breakout layout, cable gauge, length and the vendor's supported port combination.
For example, NVIDIA's 100G-PAM4 copper guide describes straight passive assemblies up to 2 m and some splitters up to 3 m, with longer active-copper options. Its interconnect specifications distinguish 100G-PAM4 and 200G-PAM4 product families. A portfolio-wide maximum is not a rating for every SKU. Cables, Transceivers and the Optics Power Budget explains the electrical and optical paths.
In the hypothetical row, suppose all 22 failed links use one older passive assembly on the longer runs, while the 42 working links use a different supported assembly. That makes compatibility or reach the leading hypothesis. It does not prove length is the cause: part number, firmware compatibility or a manufacturing batch may be correlated with the same group.
Hypothesis: one assembly type explains the 22 failed links.
- Record both endpoints
Save mode, part number, serial, state and counter deltas before moving parts.
- Substitute one supported assembly
Keep ports and configuration fixed so the comparison isolates the assembly.
- Check whether the failure moves
Repeat on compatible known-good ports, then test sustained traffic after the repair.
Gather evidence from the correct layer
On InfiniBand, ibstat reports adapter/port state and negotiated rate; fabric tools such as ibdiagnet can identify topology and link issues. On Ethernet, use the switch's port diagnostics and supported host ethtool queries. Vendor tools can expose cable inventory, lane diagnostics and physical error measurements. Select commands for the installed protocol, adapter and tool version rather than pasting an InfiniBand checklist onto an Ethernet fabric.
For each link, record:
- Administrative, physical and logical state at both ends, including the expected and negotiated mode.
- Cable/module part number, serial, rated reach, supported firmware and breakout/FEC configuration.
- Timestamped error, recovery and link-down counter deltas, plus temperature and optical measurements where supported.
- Whether the failure follows a cable, one endpoint, a switch group, a rail or a firmware version.
A PCIe inspection can help when the NIC itself is missing or underperforming, but PCIe width and fabric link width are different. lspci -vv -s <device-address> exposes the selected device's LnkCap and LnkSta; filtering the whole output for a vendor name does not verify negotiated width. These are diagnostic examples, not commands executed against hardware here.
Change one variable and check the result
Choose an affected link and a known-good compatible assembly. Preserve the baseline before replacing the assembly, then repeat the same state and counter checks. If the failure follows the assembly across known-good compatible ports, that supports a cable or module fault. If it stays with the port, investigate configuration and the endpoint. Keep the test mapping explicit; an uncontrolled sequence of swaps can hide which change helped.
A passive DAC has integrated cable ends, so “swap the transceiver but keep the DAC” is not the same operation as replacing a pluggable optical module. For optical links, inspect and clean connectors according to the vendor's procedure and check the supported fiber and optical budget. A link becoming active is only the first acceptance check: run sustained traffic and examine error and retraining rates before returning it to service. NVLink and Fabric Faults discusses why counter changes need context.
Size the impact from the actual topology
Affected links: 22 / 64 = 34.375%
If each repaired optical link needs two 15 W modules:
22 × 2 = 44 modules
44 × 15 W = 660 W of module power
sanity: this is an assumed module budget, not the net increase over the old assemblies.
The number of unusable GPUs does not follow directly from the number of dead links. Redundant paths, NIC sharing and job placement determine the impact. If 22 distinct eight-GPU nodes really must be drained, that is 176 GPUs. At an assumed allocation value of $2.50 per GPU-hour, a day represents $10,560 of affected capacity. It is not automatically lost revenue or an avoidable cash charge; some work may move elsewhere.
Replace the incompatible assemblies or correct the port configuration once the controlled test identifies the cause. If supported assemblies fail in the same ports, reverse the cable-only hypothesis and examine the endpoints. The permanent fix is a port-to-port cable plan with exact supported SKUs, run lengths, modes and acceptance results.
What interviewers probe next
- “Does flapping rule out a configuration problem?” No. State transitions narrow the investigation but do not uniquely identify the cause.
- “Do random failures prove bad parts?” No. Patterns guide tests; defects can cluster and configuration problems can appear scattered.
- “What would you order?” A vendor-supported assembly for each actual endpoint pair, lane mode and distance, including its power and cooling requirements.
Common mistakes
- Applying one generation's maximum reach to every 800G product.
- Treating a clean length boundary as conclusive evidence without a controlled test.
- Clearing counters or moving parts before recording the baseline.
- Equating 22 failed links with 22 failed nodes.
Key takeaways
- Classify link behavior, but keep overlapping causes in the investigation.
- Match exact assemblies and lane modes; “800G” alone is insufficient.
- Verify both recovery and sustained link health, then update the cable map with the tested configuration.