topology

Follow a host buffer across a NUMA boundary, calculate the cost with declared bandwidths, and align CPU workers and memory with the GPU that consumes the data.

Compare the same benchmark on a healthy node, then follow the evidence through NCCL transport choices, device topology and fabric health. Half the measured bandwidth is a symptom, not a unique diagnosis.

Count endpoints, allocate every switch port once, and compare a common fabric with independent rails before adding a third tier.

Separate the all-reduce result from NCCL execution choices, and compare latency models without treating them as universal selection rules.

Follow corresponding GPU positions through a rail, including the leaf and spine links needed when one rail outgrows a switch.

A bandwidth regression narrows the search only after the experiment is defined. Work through rank-count normalization, controlled path tests and the difference between collective time and training progress.

Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.

Use actual rank placement and measured job duration to decide whether waiting for locality beats starting immediately.

DRA gives the scheduler device claims and attributes to work with. Compare that with extended-resource counts, then check the deployed driver and controller support before promising sharing, partitioning or gang placement.