Advanced

Memory-Bound vs Compute-Bound Kernels

Use a worked timing example to distinguish a bandwidth ceiling from a measured bottleneck. Learn which profiler evidence to inspect and why decode, prefill and a whole serving step need separate diagnoses.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Roofline Model Arithmetic Intensity by Operation Prefill vs Decode Kernel Fusion Bandwidth-Bound Decode Throughput Profiling with Nsight

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureStart with bytes when sizing low-batch decode12 min→Foundations of AI InfrastructureFind a performance limit before choosing a fix14 min→Kernels and the MachineA warp is the unit of execution, and every kernel bug starts there14 min→

PRACTICE THIS IN REAL QUESTIONS

GPU & Accelerator ArchitectureIs LLM decode memory-bound or compute-bound? Show me the arithmetic that decides it.→LLM Inference & ServingWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?→GPU & Accelerator ArchitectureExplain arithmetic intensity and the roofline model. Where is the ridge point on an H100, and what does it tell you about a kernel?→Napkin Math, Cost & CapacityHow big can the decode batch get before the step becomes compute-bound?→Napkin Math, Cost & CapacityHow long does prefill take for an 8k-token prompt on a 70B model?→GPU & Accelerator ArchitectureAn Nsight summary reads DRAM 90%, SM 30%, occupancy 45%, L2 hit rate 20%. What is the bottleneck?→