21How do you overlap host-to-device transfers with compute, and what has to be true before the overlap actually happens?▼mediumNewNVIDIALambda4 replies◆ premiumChunk the work, put copies and kernels on different streams, and the total stops being copy time plus compute time. The pipeline arithmetic that says how many chunks are enough, the three conditions without which the calls run in sequence anyway, and the two levels of asynchrony that are often confused.Open full answer →
23How do you overlap communication with compute in training, and where does the overlap stop working?▼hardNewNVIDIAMeta4 replies◆ premiumBucketed all-reduce in backward, FSDP's one-layer-ahead prefetch and split-GEMM tensor parallelism hide most of the bytes, and none of it works below a certain number of tokens per GPU. The ratio that decides it, the SMs the NCCL kernels take, and the tail that is never hidden.Open full answer →
17Why do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?▼mediumNewNVIDIAMetaxAI4 replies○ sign inThe number comes from one requirement: the gradient reduction has to finish inside the backward pass that produces it. Working that requirement backward gives a bandwidth per GPU, and the answer lands near the port speed the industry ships, which is not a coincidence.Open full answer →
