activation memory

With p stages and m micro-batches the pipeline idles (p−1)/m of its compute time, and the obvious fix, more micro-batches, buys back throughput with activation memory. 1F1B caps that memory at p micro-batches instead of m. The derivation, the numbers for PP16, and the schedules that go further.

At 128k tokens one layer's activations are 73 GB per sequence and attention grows with the square of the length. The arithmetic that forces the sequence onto sixteen GPUs, what a ring of KV chunks costs per layer, why the communication hides, and where Ulysses and Megatron sequence parallelism fit around it.