22Why do tensor cores accumulate in fp32 when the inputs are bf16, and why do optimizers keep fp32 master weights?▼mediumNewNVIDIAMeta4 replies◆ premiumA float format's resolution is relative, so the damage depends on how large a running total grows against the terms being added. The measured error for a 4,096-term dot product in each direction, the weight update that vanishes entirely, and why bf16 removed loss scaling but not master weights.Open full answer →
16Explain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?▼mediumNewNVIDIAMeta4 replies○ sign infp16 loses gradients below 6e-8 and overflows above 65,504; bf16 has fp32's range and three digits of precision. The arithmetic that shows why an update of 3e-5 vanishes into a bf16 weight, where each of the 16 bytes per parameter comes from, and what fp8 block scaling adds on top.Open full answer →
