TL;DR: All three are 1 sign bit plus a split of the remaining bits between exponent (range) and mantissa (precision). fp16 is 5 + 10: max 65,504, relative step 0.1% (2^-10; the worst rounding error is half that), so it overflows on activations and underflows on gradients and needs loss scaling. bf16 is 8 + 7: the same range as fp32 with a 0.8% step (2^-7), which is why it became the default for training and serving. fp8 E4M3 is 4 + 3 (max 448, 6% step) and E5M2 is 5 + 2 (max 57,344, 12% step); E4M3 goes on weights and activations in the forward pass where precision matters, E5M2 on gradients where range matters, and both need per-tensor or per-block scaling because 4 or 5 exponent bits cannot cover a tensor's dynamic range on their own.
How to approach it
Draw the bit layouts and derive the two numbers that matter for each format, the largest representable value and the relative spacing, from the exponent and mantissa widths. Then map the two failure modes (overflow and underflow from too few exponent bits, rounding noise from too few mantissa bits) onto the tensors that exhibit them: activations, weights, gradients, optimizer state. Assign each format from that, and say what scaling machinery each needs. Close with the decision for a new training run and for a serving deployment.
A strong answer
A typical situation: a training run in fp16 produces NaN at step 4,000, the team adds loss scaling, and it works. The same run in bf16 needs no scaling at all, and the reason is one subtraction on the exponent bits.
Numerics: FP32, BF16, FP8 and FP4 all follow the same recipe: value = (−1)^sign × 2^(exponent − bias) × (1 + mantissa ÷ 2^m). The exponent bits set the range, the mantissa bits set the precision, and each format spends its budget differently.
format sign exp mant bias max value min normal relative step (2^-m)
fp32 1 8 23 127 3.4e38 1.2e-38 1.2e-7 (0.00001%)
fp16 1 5 10 15 65,504 6.1e-5 9.8e-4 (0.1%)
bf16 1 8 7 127 3.4e38 1.2e-38 7.8e-3 (0.8%)
fp8 E4M3 1 4 3 7 448 1.6e-2 1.25e-1 (12.5%)
fp8 E5M2 1 5 2 15 57,344 6.1e-5 2.5e-1 (25%)
derivations:
fp16 max = (2 − 2^-10) × 2^(30 − 15) = 1.999 × 32,768 = 65,504
bf16 max = (2 − 2^-7) × 2^(254 − 127) ≈ 2 × 2^127 = 3.4e38, same as fp32 by construction
E4M3 max = 1.75 × 2^8 = 448 (the all-ones exponent is used for values, NaN is one code)
E5M2 max = 1.75 × 2^15 = 57,344 (keeps IEEE-style inf and NaN)
relative step: the gap between adjacent values is 2^-m of the value at the bottom of each
binade, so the worst-case rounding error is half that: 0.05% fp16, 0.4% bf16, 6% E4M3, 12% E5M2
sanity: bf16 is fp32 with the low 16 bits cut off; converting is a truncation, which is why it
was the cheap thing to build first
Now the failure modes. Too few exponent bits cause overflow and underflow. In fp16, an attention logit or a pre-normalization activation in a large model can exceed 65,504 and becomes inf, and a gradient of 1e-6 (common in later layers of a deep network) falls below the 6e-5 normal range into subnormals or flushes to zero. Loss scaling exists for exactly this: multiply the loss by 2^k so gradients land in fp16's representable band, then divide the fp32 master gradients by 2^k before the update, with dynamic adjustment when an inf appears. bf16 has the fp32 exponent, so neither failure happens and loss scaling disappears; that one property is why bf16 won.
Too few mantissa bits cause rounding noise. A bf16 weight update of 1e-4 on a weight of 1.0 is below the 0.8% step and rounds to nothing, which is why mixed precision keeps an fp32 master copy of the weights and does the update there. Forward activations and the matmul inputs tolerate 0.4% error well because the products are accumulated in fp32 inside the tensor core and the noise averages out.
fp8 pushes both failure modes to the point where the format cannot stand alone. With a 4-bit exponent, E4M3 spans 2^-6 to 448, about 15 binades; a single weight tensor's values commonly span more than that when outliers are included. So every fp8 tensor carries a scale factor: the tensor (or a 128-element block of it, in per-block schemes) is multiplied by a scale chosen so its maximum lands near 448, stored in fp8, and the scale is stored in fp32 alongside. The tensor core takes the fp8 operands and applies the scales to the fp32 accumulator. Delayed scaling uses the amax of the previous few iterations; per-block scaling computes it on the fly. Either way the scale is what gives fp8 its effective range, and the 3 mantissa bits are what limit its precision.
Why two fp8 formats and where each goes:
| Tensor | Needs | Format | Why |
|---|---|---|---|
| Weights (forward) | precision; range is narrow after scaling | E4M3 | 3 mantissa bits, 6% step; scaling handles range |
| Activations (forward) | precision, with outlier handling | E4M3 with per-block or per-channel scale | outliers in a few channels would otherwise set the scale for everything |
| Gradients (backward, dL/dx and dL/dW inputs) | range; values span many orders | E5M2 | 5 exponent bits, same range as fp16; 12% step is tolerated because gradients are noisy anyway |
| Optimizer state, master weights | full precision | fp32 (or bf16 with stochastic rounding, carefully) | accumulated updates are far below any 8-bit step |
| Accumulators, softmax, layernorm statistics | full precision | fp32 | sums of thousands of products |
The practical assignments:
- Training a new model today: bf16 for weights-in-compute, activations and gradients, fp32 master weights and optimizer state, no loss scaling. Add fp8 (E4M3 forward, E5M2 backward, per-block scaling) for the large linear layers once the recipe is validated against a bf16 run on the same data, keeping attention, normalization and the last layers in bf16. The memory saving is real: fp8 activations halve the activation memory that dominates at long sequence lengths, and the fp8 tensor-core peak is 2x bf16.
- Serving: fp8 E4M3 weights and, where the engine supports it, fp8 KV cache and fp8 activations for the GEMMs. Weights halve (141 GB to 71 GB for a 70B model), decode step time halves with them, and accuracy loss is small with per-channel weight scales and per-token activation scales. fp16 has no role on modern hardware unless a kernel library only ships fp16.
The reversal condition, and the thing that makes Tensor Cores and Matrix Units refuse to help: a model whose activations have extreme outlier channels (some older architectures, or models trained without the normalization choices that suppress them) can lose measurable accuracy in fp8 even with per-block scaling; the test is a perplexity and task-eval comparison against bf16, not a belief.
What interviewers probe next
- "Why accumulate in fp32 if inputs are fp8?" A dot product of 8,192 terms sums values that individually carry 6% error; in fp32 the accumulation adds almost nothing to that, in fp16 or bf16 the sum itself would lose bits as it grows. The tensor core accumulates in fp32 for free.
- "What is tf32?" fp32's exponent with a 10-bit mantissa, used inside tensor cores for fp32 matmuls: same range as fp32, fp16's precision, 8x the fp32 throughput.
- "Does fp8 KV cache hurt?" Keys are more sensitive than values because they enter the softmax; per-head scales and E4M3 keep the loss small on most models, but long-context retrieval tasks are where it shows first.
- "Why do gradients tolerate 12% error?" Stochastic gradient descent already carries batch noise far larger than that; the update direction survives coarse quantization as long as it does not flush to zero, which is the range problem, not the precision problem.
Common mistakes
- Saying bf16 "has less precision than fp16, so it is worse" without noting that the range difference is what caused every fp16 training failure.
- Presenting fp8 as a drop-in dtype; without scales it is unusable, and which scaling scheme is in use decides the accuracy.
- Assigning E5M2 to weights "because more range is safer": the precision loss at 2 mantissa bits is visible in model quality.
- Forgetting that the optimizer state stays fp32, and then computing training memory as 2 bytes per parameter.
Key takeaways
- Range comes from exponent bits, precision from mantissa bits: fp16 5+10 (max 65,504), bf16 8+7 (max 3.4e38, 0.8% step), E4M3 4+3 (max 448), E5M2 5+2 (max 57,344).
- bf16 removed loss scaling by matching fp32's exponent; that is why it is the default.
- fp8 needs a scale per tensor or per block; E4M3 for forward weights and activations, E5M2 for gradients, fp32 for accumulation and optimizer state.
- Serving in fp8 halves weight bytes and decode step time; validate against bf16 with perplexity and task evals before trusting it.