mediumNVIDIAAMDMicron

Explain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.

Derive HBM bandwidth from interface width and pin rate, then connect it to a model that actually fits. Separate raw interface capacity from measured decode time.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Derive HBM bandwidth from interface width and pin rate, then connect it to a model that actually fits. Separate raw interface capacity from measured decode time.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Memory-Bound vs Compute-Bound KernelsUse a worked timing example to distinguish a bandwidth ceiling from a measured bottleneck. Learn which profiler evidence to inspect and why decode, prefill and a whole serving step need separate diagnoses.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

Foundational

🧩 GPU & Accelerator Architecture

GPU Memory HierarchyTrace storage scope and data reuse from registers through shared memory, caches and HBM. Worked register and traffic examples explain why a source-level load is not automatically an HBM transfer.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache saves attention keys and values from processed tokens. Follow one decode step, calculate its memory cost, and see why longer conversations reduce the number of requests a GPU can hold.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat actually changes with Blackwell and the NVL72 rack, and what does it do to how you would serve a large MoE model?Next in this trackYou have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?Next in this trackExplain how a systolic array computes a matrix multiply. Why is it efficient, and why does it want big square tiles?

FEDITOR'S NOTE

Separate memory capacity from bandwidth. Check that the model fits, derive the weight-read floor, and use the trace to decide whether memory traffic is the useful target.

DISCUSSION · 0

No comments yet — be the first to share your approach.