Advanced

FlashAttention Internals

FlashAttention computes exact attention in tiles without storing the full score or probability matrix in device memory. Follow the running softmax state, understand why inputs can still be reread, and distinguish the mechanisms introduced in the first three papers.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Kernel Fusion Shared Memory and Bank Conflicts Tensor Cores and Matrix Units Attention Variants: MHA, GQA, MQA and MLA Prefill vs Decode CUTLASS and Tensor Core Kernels

LESSONS THAT TEACH THIS

Kernels and the MachineA tiled matrix multiply connects ownership, reuse and synchronization18 min→Kernels and the MachineFusion removes traffic that tiling cannot19 min→Kernels and the MachineIO-aware attention is a memory-traffic algorithm, not a faster matmul29 min→

PRACTICE THIS IN REAL QUESTIONS

CUDA, Triton & Kernel EngineeringExplain FlashAttention. Why is it called IO-aware, and what does it actually save?→CUDA, Triton & Kernel EngineeringWhat changed between FlashAttention 1, 2 and 3, and why did each change buy what it did?→CUDA, Triton & Kernel EngineeringYour fused attention kernel matches the reference at 512 tokens and drifts at 8,000. Find the bug.→Napkin Math, Cost & CapacityA kernel does 4 TFLOP and moves 40 GB in one call. On an H100, is it memory-bound or compute-bound?→CUDA, Triton & Kernel EngineeringExplain shared memory bank conflicts with the bank arithmetic, show a kernel that has them, and fix it with padding.→LLM Inference & ServingWhat is the KV cache, and why does it keep growing while a request is being served?→