memory bound
AI infra interview questions tagged memory bound, across every topic.
3 questions · 2 unlocked for you
Concepts behind "memory bound"
The curriculum that explains the ideas these questions test.
Foundational
Roofline ModelThe roofline plots a kernel's attainable throughput against its arithmetic intensity, FLOPs per byte moved from memory. Below the ridge point (peak FLOPS divided by memory bandwidth, about 295 on an H100 in bf16) a kernel is memory-bound and no amount of clever code reaches the peak; above it, compute is the limit. One picture explains why decode runs at under 1% of peak and why fusion and batching are the two levers that move it.🧩 GPU & Accelerator Architecture
Foundational
Kernel FusionAn elementwise or reduction kernel does a few FLOPs per byte and runs at HBM speed, so a chain of five of them costs five trips through HBM for work that needs one. Fusion collapses the chain into a single kernel that keeps intermediates in registers. It is the first lever for anything memory-bound, and knowing what it cannot fix (weight reads in decode, the GEMMs themselves) is what the interview is really testing.⚡ Kernels & Compilers
Foundational
Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.🚀 Inference & Serving
Advanced
Memory-Bound vs Compute-Bound KernelsEvery kernel is limited by one of two walls: how fast bytes arrive from HBM, or how fast the tensor cores can multiply. Which wall applies is decided by arithmetic intensity against the ridge point, and the two regimes need opposite fixes. Decode, LayerNorm and softmax are memory-bound; prefill GEMMs are compute-bound; the interview question is which one you are looking at and what you would do about it.🧩 GPU & Accelerator Architecture🔒 Premium
Advanced
Profiling with NsightNsight Systems answers where wall-clock time goes across CPU, kernels and copies; Nsight Compute answers why one kernel is slow, from hardware counters. The skill interviewers test is the order: timeline first, then the Speed of Light section, then the two or three metrics that name the bottleneck, so that a memory-bound kernel is recognized from its profile in under a minute and the fix is bytes, not occupancy.⚡ Kernels & Compilers🔒 Premium