Why do tensor cores accumulate in fp32 when the inputs are bf16, and why do optimizers keep fp32 master weights?

A float format's resolution is relative, so the damage depends on how large a running total grows against the terms being added. The measured error for a 4,096-term dot product in each direction, the weight update that vanishes entirely, and why bf16 removed loss scaling but not master weights.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Every number format is a trade between range (exponent bits), precision (mantissa bits) and throughput (fewer bits, more values per cycle through the tensor cores). bf16 won training because it keeps fp32's range; fp8 splits into E4M3 for precision and E5M2 for range and needs scaling factors; fp4 needs block scaling and careful outlier handling. Knowing which format goes where, and why accumulation stays fp32, is what the numerics question is really asking.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQOpen-weights models now ship pre-quantized, and the format is part of the release rather than something you choose afterwards. Block-scaled FP8 gives one byte per parameter with a scale per tile. MXFP4 gives about 0.53 bytes by pairing four-bit values with a shared exponent every 32 elements. Integer schemes like AWQ reach similar sizes with a different error profile. What decides a deployment is not which is most accurate in the abstract but which one the model was released and evaluated in, and which one your engine and hardware can execute natively.

Foundational

⚡ Kernels & Compilers

CUDA Programming ModelCUDA splits a program into a host that allocates, copies and enqueues work, and a device that runs thousands of identical threads organized as a grid of blocks. Getting the split right, and knowing that a launch returns before the kernel runs, decides whether your first live-coding kernel produces a correct number or a silent zero.

Core

⚡ Kernels & CompilersSign in

Memory CoalescingA warp's 32 threads issue one memory request together, and the hardware serves it in 32-byte sectors. Coalescing is arranging addresses so those sectors are full of bytes the warp will use. It decides whether a bandwidth-bound kernel moves at the HBM rate or at an eighth of it, and it is the pattern NVIDIA's trace-classification interview question tests.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackIn an int8 or fp8 GEMM, what exactly gets scaled, where does the dequantization happen, and what breaks if you get it wrong?Next in this trackA mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?Next in this trackSelect the top k logits from a 128,000-token vocabulary on the GPU. What shape does the kernel take and why not just sort?

FEDITOR'S NOTE

Scored on explaining accumulation error as a ratio between the running sum and the addend rather than as a function of length alone, on the master-weight argument, and on knowing that bf16 removes loss scaling but not fp32 accumulation.

DISCUSSION · 0

No comments yet — be the first to share your approach.