Core

Arithmetic Intensity by Operation

The roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.

a free account unlocks the core curriculum tier · no card

RELATED CONCEPTS

Roofline Model Memory-Bound vs Compute-Bound Kernels Bandwidth-Bound Decode Throughput FlashAttention Internals Kernel Fusion Tensor Cores and Matrix Units

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureEvery performance question is bound by something before it is anything else14 min→Foundations of AI InfrastructurePrefill is a matrix multiply; decode is a memory read wearing a matmul's clothes14 min→

PRACTICE THIS IN REAL QUESTIONS

Napkin Math, Cost & CapacityHow big can the decode batch get before the step becomes compute-bound?→LLM Inference & ServingWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?→Napkin Math, Cost & CapacityA kernel does 4 TFLOP and moves 40 GB in one call. On an H100, is it memory-bound or compute-bound?→GPU & Accelerator ArchitectureIs LLM decode memory-bound or compute-bound? Show me the arithmetic that decides it.→Napkin Math, Cost & CapacityA vendor claims 10,000 tokens per second per GPU. Sanity-check it.→Napkin Math, Cost & CapacityHow long does prefill take for an 8k-token prompt on a 70B model?→