accumulation

A float format's resolution is relative, so the damage depends on how large a running total grows against the terms being added. The measured error for a 4,096-term dot product in each direction, the weight update that vanishes entirely, and why bf16 removed loss scaling but not master weights.

The matmul runs on tiny formats and the accumulator does not, so every low-precision GEMM is really a scaled integer or float product with a correction applied on the way out. Where the scales come from, the three granularities and what each costs, and why one of them needs a periodic promotion into fp32.

Three mantissa bits and a range of 448 make fp8 unusable with one scale per tensor. The arithmetic of what an outlier channel destroys, the 128-element block scaling and fp32 promotion that the DeepSeek-V3 report used, the ops that stayed in bf16, and what the 2× peak bought in practice.