cuda graphs

Four stages: a bytecode interpreter captures a graph with guards, an autograd pass splits forward from backward, a compiler emits Triton for everything but the matmuls, and a mode that replays the step as one launch. What each buys, and the two failures that silently give it back.

At batch 1 a decode step is a few milliseconds of GPU work behind several hundred kernel launches, and the CPU cannot issue them fast enough. The arithmetic showing the GPU starve, what replay changes, and the four constraints capture imposes on memory.

The bound assumes weights stream contiguously and nothing else costs time, and at batch one both assumptions fail badly. Four terms that make up the gap, why sparse models suffer most, and the two fixes that recover most of it.

Graphs remove the per-kernel launch cost, which is a large share of a decode step at low batch and a small one at high batch, so whether it matters is a question about your operating point. The arithmetic, the four reasons an engine turns them off, and the measurement that settles it.