bf16

Every format is a trade between how big a number can be and how finely it is spaced. The bit layouts, the largest and smallest values derived from them, why fp16 needed loss scaling and bf16 did not, and why fp8 comes in two flavors with a scale factor attached.

A float format's resolution is relative, so the damage depends on how large a running total grows against the terms being added. The measured error for a 4,096-term dot product in each direction, the weight update that vanishes entirely, and why bf16 removed loss scaling but not master weights.

fp16 loses gradients below 6e-8 and overflows above 65,504; bf16 has fp32's range and three digits of precision. The arithmetic that shows why an update of 3e-5 vanishes into a bf16 weight, where each of the 16 bytes per parameter comes from, and what fp8 block scaling adds on top.