flashattention

Standard attention writes an N by N score matrix to HBM and reads it back twice. At 8k context that is 134 MB per head. The traffic arithmetic before and after tiling, the online-softmax identity that makes one pass possible, where the kernel lands on the roofline, and why the memory saving matters more than the speed.

Each version fixed a different bottleneck: version 1 fixed memory traffic, version 2 fixed non-matmul work and parallelism, version 3 fixed the fact that softmax and matmul were waiting for each other on Hopper. The arithmetic that shows why a softmax costing one percent of the FLOPs can cost sixty percent of the time.

A bug that scales with the number of tiles is invisible in a unit test that fits in one tile. The three candidates that produce exactly this signature, the measurement that separates them, and a test design that would have caught all three before the kernel shipped.