Core

Arithmetic Intensity by Operation

Count the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

This concept is included with a free account. Sign in to continue reading.

free account · no card needed

RELATED CONCEPTS

Roofline Model Memory-Bound vs Compute-Bound Kernels Bandwidth-Bound Decode Throughput FlashAttention Internals Kernel Fusion Tensor Cores and Matrix Units

LESSONS THAT TEACH THIS

Foundations of AI InfrastructureFind a performance limit before choosing a fix14 min→Foundations of AI InfrastructureWeight reuse explains why prefill and decode need different tuning19 min→Kernels and the MachineReuse is the only thing that moves a kernel up the roofline16 min→

PRACTICE THIS IN REAL QUESTIONS

Napkin Math, Cost & CapacityHow big can the decode batch get before the step becomes compute-bound?→LLM Inference & ServingWhy do prefill and decode behave so differently, and why does that matter for the hardware you serve on?→Napkin Math, Cost & CapacityA kernel does 4 TFLOP and moves 40 GB in one call. On an H100, is it memory-bound or compute-bound?→GPU & Accelerator ArchitectureIs LLM decode memory-bound or compute-bound? Show me the arithmetic that decides it.→Napkin Math, Cost & CapacityA vendor claims 10,000 tokens per second per GPU. Sanity-check it.→Napkin Math, Cost & CapacityHow long does prefill take for an 8k-token prompt on a 70B model?→