Advanced

Memory-Bound vs Compute-Bound Kernels

Every kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Roofline Model Arithmetic Intensity by Operation Prefill vs Decode Kernel Fusion Bandwidth-Bound Decode Throughput Profiling with Nsight

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureA GPU is a bandwidth machine that happens to do arithmetic12 min→Foundations of AI InfrastructureEvery performance question is bound by something before it is anything else14 min→

PRACTICE THIS IN REAL QUESTIONS

GPU & Accelerator ArchitectureIs LLM decode memory-bound or compute-bound? Show me the arithmetic that decides it.→LLM Inference & ServingWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?→GPU & Accelerator ArchitectureExplain arithmetic intensity and the roofline model. Where is the ridge point on an H100, and what does it tell you about a kernel?→Napkin Math, Cost & CapacityHow long does prefill take for an 8k-token prompt on a 70B model?→Napkin Math, Cost & CapacityHow big can the decode batch get before the step becomes compute-bound?→GPU & Accelerator ArchitectureAn Nsight summary reads DRAM 90%, SM 30%, occupancy 45%, L2 hit rate 20%. What is the bottleneck?→