all reduce

Normalize link directions, count collective bytes, and see why a gradient reduction can outlast its real overlap window.

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Trace a chunk, derive both traffic phases, and state the endpoint model behind bandwidth optimality. A worked example shows why nearly constant bytes do not imply constant completion time.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Count gradient traffic with the actual dtype, convert one-direction rates, and expose the bucket-readiness constraint that an average bandwidth division misses.

Derive ring all-reduce traffic, match directional bytes to bandwidth, and check memory feasibility before estimating the communication exposed on a training step.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.

Track actual values across ranks, then count bytes with an explicit buffer definition. Reduction combines values; gathering and all-to-all change ownership.

Derive bandwidth from actual per-rank traffic and readiness deadlines, showing why bucketing cannot erase a capacity shortfall.