Advanced

FlashAttention Internals

Standard attention writes the N x N score matrix to HBM and reads it back, which makes it memory-bound and quadratic in memory. FlashAttention tiles Q, K and V through shared memory, keeps a running max and sum so the softmax never needs the full row, and recomputes scores in the backward pass. Knowing the online-softmax rescale, why FlashAttention-2 flipped the loop order, and what FlashAttention-3 overlaps on Hopper is the difference between naming the paper and being able to write the kernel.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Kernel Fusion Shared Memory and Bank Conflicts Tensor Cores and Matrix Units Attention Variants: MHA, GQA, MQA and MLA Prefill vs Decode CUTLASS and Tensor Core Kernels

PRACTICE THIS IN REAL QUESTIONS

CUDA, Triton & Kernel EngineeringExplain FlashAttention. Why is it called IO-aware, and what does it actually save?→CUDA, Triton & Kernel EngineeringWhat changed between FlashAttention 1, 2 and 3, and why did each change buy what it did?→CUDA, Triton & Kernel EngineeringYour fused attention kernel matches the reference at 512 tokens and drifts at 8,000. Find the bug.→Napkin Math, Cost & CapacityA kernel does 4 TFLOP and moves 40 GB in one call. On an H100, is it memory-bound or compute-bound?→LLM Inference & ServingWhat is the KV cache, and why does it keep growing while a request is being served?→CUDA, Triton & Kernel EngineeringExplain shared memory bank conflicts with the bank arithmetic, show a kernel that has them, and fix it with padding.→