mediumNewNVIDIAAMDMicron

Explain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.

HBM stacks DRAM dies on top of each other and wires them to the GPU through a silicon interposer with a 1,024-bit bus per stack. That design sets how much bandwidth and capacity a card can have, why the two scale together, and why they have grown more slowly than FLOPS across three generations.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsEvery kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationThe roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.

Foundational

🧩 GPU & Accelerator Architecture

GPU Memory HierarchyA GPU has four places a byte can live, and they differ by a thousandfold in bandwidth: registers, shared memory on the SM, a chip-wide L2, and HBM off-chip. Almost every kernel optimization is a decision about which level a value is read from and how many times. Knowing the sizes and bandwidths for an H100 cold is what lets you say why a kernel is slow before you profile it.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat actually changes with Blackwell and the NVL72 rack, and what does it do to how you would serve a large MoE model?Next in this trackYou have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?Next in this trackExplain how a systolic array computes a matrix multiply. Why is it efficient, and why does it want big square tiles?

FEDITOR'S NOTE

Scored on a correct physical picture of HBM (stacks, interposer, wide bus), the per-stack arithmetic, and the generation-over-generation ratio that shows bandwidth falling behind compute.

DISCUSSION · 0

No comments yet — be the first to share your approach.