nccl

Compare host-staged and direct GPU-to-NIC transfers, count traffic at each boundary, and check transport selection before blaming a slow collective on missing GPUDirect.

Diagnose a multi-node regression by comparing host and GPU transfer paths, active registration mechanisms and the tested software stack.

Trace a chunk, derive both traffic phases, and state the endpoint model behind bandwidth optimality. A worked example shows why nearly constant bytes do not imply constant completion time.

Use per-rank arrival times and controlled substitutions to explain a slow step without mistaking collective wait for network transfer.

Read a watchdog timeout without blaming the reporting rank. Compare group-scoped traces, separate missing evidence from mismatched calls, and distinguish CPU submission from GPU completion.

The nodes joined the cluster, but the first request hangs. Trace how far it got, check worker assignments, and reproduce the blocked operation on the same communication path.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

Separate the all-reduce result from NCCL execution choices, and compare latency models without treating them as universal selection rules.

Follow corresponding GPU positions through a rail, including the leaf and spine links needed when one rail outgrows a switch.

Separate NCCL logging, device selection and execution overrides from PyTorch failure handling. Work through a faster collective that makes the job slower, and learn which timeout actually fired.

Distinguish GPU-memory capability, registration and selected transport. Compare direct and staged paths without relying on a half-speed rule or one log substring.

A fabric address, an RDMA device and a working GPU-memory registration path are separate requirements. Verify each inside the pod, then test the actual GPU-to-NIC placement and collective behavior.

Check useful progress before interpreting GPU activity. Run a bounded trace exercise that exposes a rank-conditional call, equal-count ordering errors and matching shapes with the wrong tensor meaning.