How do you overlap communication with compute in training, and where does the overlap stop working?

Bucketed all-reduce in backward, FSDP's one-layer-ahead prefetch and split-GEMM tensor parallelism hide most of the bytes, and none of it works below a certain number of tokens per GPU. The ratio that decides it, the SMs the NCCL kernels take, and the tail that is never hidden.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧭 Ownership & Judgment

Talking About Cost and Capacity with LeadershipInfrastructure engineers are asked to justify large numbers to people who do not share their vocabulary, and the conversations go wrong in predictable ways: a technical objection with no alternative, a forecast with no assumptions, or a cost quoted in a unit the listener cannot act on. What works is a small number of costed options, a stated recommendation, the decision needed by a date, and every figure expressed in whatever the listener actually controls.

Foundational

🧭 Ownership & Judgment

Escalation That WorksEscalation has a reputation as a political act because most of it is done badly: a problem handed upward with no options and an implicit request that someone else choose a side. Done well it is a one-page artifact with two or three costed options, a recommendation, the decision needed, a date, and what you will do by default if no answer arrives. That last line is what converts a message into a decision, and it is the part almost everyone omits.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?Next in this trackFine-tune a 70B on one 80 GB card. What do NF4 and double quantization actually buy, and where does DoRA change the arithmetic?Next in this trackDesign the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?

FEDITOR'S NOTE

Scored on deriving the tokens-per-GPU threshold below which overlap cannot hide FSDP traffic, on naming the three mechanisms at their level, and on the limits: SM contention, memory for prefetch, and the exposed tail.

DISCUSSION · 0

No comments yet — be the first to share your approach.