Why do tensor cores accumulate in fp32 when the inputs are bf16, and why do optimizers keep fp32 master weights?

Watch a low-precision sum stop changing and a small weight update round away. Separate storage precision, accumulation, optimizer state and dynamic range before choosing a mixed-precision training policy.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

Roofline calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Core

🧩 GPU & Accelerator ArchitectureSign in

Numerics: FP32, BF16, FP8 and FP4Floating-point formats trade range and precision for storage and compute cost. Understand exponent bits, significand bits, scaling and accumulation before choosing a format.

Advanced

⚡ Kernels & Compilers🔒 Premium

Kernel Correctness ContractsA fast GPU kernel is useful only for inputs it handles correctly. Define shapes, strides, dtypes, numerical tolerance, storage ownership and synchronization before timing a replacement.

Foundational

🧮 Open Weights & Serving Engines

Weight Formats: FP8 Blocks, MXFP4 and AWQDerive the storage cost of low-precision tensors, including scales, zero points and higher-precision exceptions. Distinguish formats from quantization methods, understand fused dequantization, and test whether a smaller checkpoint actually improves serving capacity without unacceptable quality loss.

Foundational

⚡ Kernels & Compilers

CUDA Programming ModelCUDA programs launch GPU kernels from host code. Thread indexing, memory access and stream ordering determine whether the result is correct.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackIn an int8 or fp8 GEMM, what exactly gets scaled, where does the dequantization happen, and what breaks if you get it wrong?Next in this trackA mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?Next in this trackSelect the top k logits from a 128,000-token vocabulary on the GPU. What shape does the kernel take and why not just sort?

FEDITOR'S NOTE

Explain representable spacing with an executed example, distinguish accumulation from operand quantization, and count the actual optimizer buffers instead of assuming every training stack uses the same master-weight layout.

DISCUSSION · 0

No comments yet — be the first to share your approach.