bf16, fp16, fp8: what is the difference at the bit level, and where does each one belong in training and serving?

TL;DR: FP16 and BF16 both use two bytes, but FP16 keeps finer spacing near one while BF16 has a much wider exponent range. FP8 halves the raw payload again and usually needs a carefully chosen scaling recipe. Storage format, multiplication format and accumulation format are separate choices. None removes the need to check overflow, underflow and model quality.

How to approach it

Explain range versus spacing with actual numbers. Name the precise FP8 variant rather than just “eight-bit.” Then separate loss scaling, quantization scaling and wider accumulation. Finish with a small numerical test and an equal-quality runtime comparison.

A strong answer

A training run overflows after switching to FP16. Another run fits in BF16 but small weight updates stop changing a stored value. These are different failures: one concerns range, the other representable spacing.

For an ordinary normalized binary floating-point value, the representation combines a sign, an exponent and a fraction. The exponent selects a scale; fraction bits choose nearby values within that scale. Zero, subnormal values and special encodings need separate rules. NVIDIA's floating-point guide explains why finite precision and rounding are part of every arithmetic result.

Near one, the next larger normalized value is 1 + 2^(-fraction_bits). This makes the precision trade visible without memorizing a vague number of decimal digits:

The FP8 names spell out the bit split: E4M3 has 4 exponent bits and 3 fraction (mantissa) bits, and E5M2 has 5 and 2. The FN suffix means finite-only, with no encoding spent on infinity.

Here is the E4M3FN spacing derived from its fraction field:

fraction bits = 3
spacing above 1 = 2^(-3) = 0.125
next value = 1 + 0.125 = 1.125
half this interval = 0.0625
sanity: eight intervals of 0.125 span [1, 2].

The table shows the gap between adjacent values near one. It is not a relative-error bound for every possible input. Under round-to-nearest, an input between two adjacent values rounds to one of them, so its absolute rounding error is at most half the gap. A tie rule decides which endpoint to use at the midpoint. Near zero, subnormal and underflow behavior change the simple relative-error picture.

For example, 1 + 0.0001 rounds back to 1 in either FP16 or BF16 with round-to-nearest. Repeating an update on an already rounded stored weight can lose changes that a higher-precision master value would retain. The training recipe may therefore keep some state or accumulation wider than its matrix inputs. This is a numerical reason for mixed precision, not a promise that one layout is required by every optimizer.

FP16's largest finite value is 65,504. BF16 has eight exponent bits, like FP32, and a largest finite value near 3.39×10³⁸, but fewer fraction bits. That wider range often reduces the need for gradient loss scaling; it does not make BF16 immune to nonfinite values or eliminate small-value rounding.

The FP8 format paper defines the E4M3 and E5M2 tradeoff. For the finite E4M3 variant used here, called E4M3FN in common tensor APIs, the maximum is 448. E5M2's maximum is 57,344. Other eight-bit variants have different special encodings; their names and limits cannot be silently interchanged.

A simple illustrative scaling calculation maps a tensor's largest absolute value 1,000 to E4M3FN's 448 with multiplier 448/1000 = 0.448. Quantize the scaled values, then interpret them using the inverse scale. Values above the calibrated range can clip; smaller values still encounter coarse spacing. Scaling shifts the range, but creates no extra fraction bits. Actual recipes may select power-of-two scales or different granularity.

Loss scaling is mainly a response to small-gradient underflow. It does not fix an activation that already overflowed in the forward pass; a larger scale can itself overflow gradients. Coordinate finite checks and skipped updates across distributed ranks. Numerics: FP32, BF16, FP8 and FP4 covers how this choice interacts with training state.

Tensor Cores and Matrix Units explains the hardware side of the format choice. Choose a supported recipe, test it against a higher-precision reference, and measure both model-quality behavior and runtime. Watch finite values, per-layer ranges, gradient norms and loss trajectories; a single short prompt or step is weak evidence. In serving, compare task-relevant quality and long-context cases too. If FP8 conversion or layout overhead cancels the traffic saving, or sensitive layers fail the quality threshold, retain a wider format there. Halving payload is not a guaranteed twofold speedup.

What interviewers probe next

• “Must forward use E4M3 and backward E5M2?” That is one recipe, motivated by different range needs. It is not a universal instruction requirement or proof of convergence.
• “Does conversion just truncate bits?” Rounding and saturation behavior depend on the operation. Name the mode used by the tested implementation.
• “Does FP32 accumulation remove quantization error?” It improves accumulation precision but cannot reconstruct information already lost in rounded inputs.
• “How much raw storage does a billion values need?” Two decimal GB in FP16 or BF16, one GB in FP8, before scales and other tensors.

Common mistakes

• Confusing the 12.5% E4M3 spacing above one with a universal rounding-error bound.
• Saying loss scaling repairs forward activation overflow.
• Treating all FP8 variants as the same format.
• Inferring model quality or speed from byte width alone.

Key takeaways

• FP16 and BF16 use equal storage but allocate their 16 bits differently.
• The next value above one is 1.125 in E4M3FN and 1.25 in E5M2.
• Scaling, multiplication and accumulation are distinct numerical decisions that need testing together.