nvlink

Normalize link directions, count collective bytes, and see why a gradient reduction can outlast its real overlap window.

Separate Blackwell chip gains from the 72-GPU domain, follow a token to its expert owners, and test placement instead of dividing model bytes by 72.

Compare the same benchmark on a healthy node, then follow the evidence through NCCL transport choices, device topology and fabric health. Half the measured bandwidth is a symptom, not a unique diagnosis.

Size adapter and full-parameter jobs before comparing RTX PRO 6000 and H100. Test memory headroom, MIG packing, software support and real communication costs.

Fabric Manager helps make an NVSwitch system ready. Learn the platform-specific roles, failure symptoms and checks that prove the communication path works.

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

Use actual rank placement and measured job duration to decide whether waiting for locality beats starting immediately.

Use the Llama 3 interruption record without turning job interruptions into device lifetime, predictable failures or a universal spare percentage.

Interpret XID codes with device and software context, separating immediate recovery from evidence-based root-cause investigation.

Read link counters with their generation-specific units and traffic exposure. Correlate changes with actual communication performance, then choose monitoring, maintenance or immediate recovery from the supported fault state.