dataloader

A two-socket GPU node has two PCIe trees, two memory pools and one inter-socket link that every wrong-sided transfer must cross. A dataloader on the far socket shares that link with seven other GPUs' traffic, and its copies run at half speed. The topology map, the arithmetic and the pinning.

Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.

Compute the step time the formula predicts, measure the one you have, and the gap is the budget to explain. The isolation order with one metric per suspect: dataloader wait, SM and tensor-core activity, HBM throughput, time in collectives, and the per-rank spread that says it is one machine.