communication

Not the data, not the weights: the gradients, once per step, in a ring that moves almost twice the model's size through every GPU. The derivation, the per-step byte count for an 8B model, and why it still hides behind the backward pass.

Four all-reduces per transformer block, on activations, on the critical path. Inside the NVLink domain they cost 8% of a step; across the NIC they cost more than the step itself. The derivation for a 70B, and the second reason TP stops that has nothing to do with the network.

Mathematically identical for a mean loss and a stateless model, and different in three ways that matter to infrastructure: activation memory, communication frequency, and any layer that looks at the batch. With the byte counts and the DDP call that makes it work.

Bucketed all-reduce in backward, FSDP's one-layer-ahead prefetch and split-GEMM tensor parallelism hide most of the bytes, and none of it works below a certain number of tokens per GPU. The ratio that decides it, the SMs the NCCL kernels take, and the tail that is never hidden.

A war story with no learning in it scores zero, however dramatic. The five beats of an incident narrative, the decision under uncertainty that is the actual subject of the question, and how to tell it when the incident was somebody else's fault.

The first move is not technical. It is deciding who runs what, because one person serially debugging three incidents is slower than three people in parallel and much slower than one person coordinating. The ordering rule, and the two things that outrank everything else.

Bad news delivered as a problem gets you a meeting. Delivered as a forecast with options, costs and a decision date, it gets a decision. The four-part structure, the projection that has to be defensible, and the mistake of waiting until you are certain.