numa

A two-socket GPU node has two PCIe trees, two memory pools and one inter-socket link that every wrong-sided transfer must cross. A dataloader on the far socket shares that link with seven other GPUs' traffic, and its copies run at half speed. The topology map, the arithmetic and the pinning.

Five causes produce the same symptom and each has a check that takes under a minute. The order to run them in, what each one eliminates, and the arithmetic showing that one slow node in a thousand costs the whole job its speed rather than a thousandth of it.