batch size

A linear layer at batch 1 runs at a third of a percent of peak and at batch 4,096 near 70%. The curve between has two regions and a sawtooth: the memory-bound slope where intensity equals M, the plateau where the tensor cores are the limit, and the dips where the tile count does not divide the SM count.

Mathematically identical for a mean loss and a stateless model, and different in three ways that matter to infrastructure: activation memory, communication frequency, and any layer that looks at the batch. With the byte counts and the DDP call that makes it work.

A small model guesses four tokens and the big one checks them in a single step. That converts idle bandwidth into tokens, and the arithmetic tells you exactly which batch size stops it working.

Tokens per second climbs with batch and per-token latency climbs with it. The curve has a knee set by the hardware ridge, and the operating point is where your latency SLO cuts it.

The bound assumes weights stream contiguously and nothing else costs time, and at batch one both assumptions fail badly. Four terms that make up the gap, why sparse models suffer most, and the two fixes that recover most of it.