TL;DR: FlashAttention-1 reduces transfers of intermediate results through high-bandwidth memory (HBM). FlashAttention-2 improves work partitioning and reduces non-matmul work. FlashAttention-3 uses Hopper's asynchronous execution to overlap data movement, matrix multiplication and softmax, and adds an FP8 path with accuracy techniques. These are different bottlenecks, not three universal speed multipliers. Match the implementation to the hardware, shape and numerical contract.
How to approach it
Compare what work each historical version removes or overlaps. Draw a dependency that can overlap and one that cannot. Then explain how you would check a version-to-version result without mixing devices, dtypes, forward/backward work or theoretical peaks.
A strong answer
An attention kernel stopped writing the full score matrix, yet its Tensor Core utilization is still low. The next improvement may come from distributing work more evenly or scheduling the remaining scalar work around matrix instructions. Repeating “it saves HBM” does not explain every later version.
This comparison covers the mechanisms introduced in the 2022, 2023 and 2024 papers. It is not a claim that version 3 is the latest available implementation, or that every current library bearing a version label has identical kernels.
Version 1: remove a large intermediate
The first step tiles the score and value calculations and maintains online normalization state. This avoids materializing the complete N×N score/probability matrices in HBM. Dense arithmetic remains quadratic. FlashAttention Internals explains the rescale and the difference between eliminated intermediates and actual input traffic.
The remaining program still has to distribute blocks across SMs, share work among warps and execute reductions, rescaling and exponentials. A kernel can avoid the biggest allocation and still spend substantial time outside its fast matrix instructions.
Version 2: distribute work and reduce coordination
FlashAttention-2 adds more sequence-dimension parallelism, which helps fill the GPU when batch and head counts supply too few blocks. Its forward work partitioning assigns different query rows to warps while sharing key/value data, reducing the need to combine partial outputs across warps. It also keeps an unnormalized output during the tile loop and divides at the end, reducing repeated non-matmul normalization work.
That does not eliminate the running-max rescale. The accumulated numerator and denominator still need a common scale. Nor does “sliced Q” mean every attention pass in every implementation uses the same partition; the forward and backward data dependencies differ.
Version 3: overlap work whose dependencies permit it
Hopper provides asynchronous matrix instructions and the Tensor Memory Accelerator for data movement. FA3 organizes producer/consumer work and pipelines attention operations so scalar softmax can overlap suitable matrix work. The sketch shows one important relationship from its two-stage approach.
Softmax for score tile j must wait for that tile's QK result. Its PV multiplication must wait for its probabilities. But QK for the next key tile does not need those probabilities, so there is useful work to overlap when the data and resources are ready. Asynchronous execution does not remove dependencies; it gives the scheduler room to satisfy them while doing other work.
A shared buffer needs two distinct notifications: data is ready for the consumer, and the consumer has finished so the producer may reuse the storage. Missing the second can corrupt results even when the first wait is correct. More buffering consumes shared memory and registers, potentially reducing residency. Occupancy and Register Pressure explains why deeper pipelines can lose on shapes that cannot amortize their overhead.
FP8 changes the accuracy question too
FA3's FP8 forward path quantizes small blocks separately and uses an orthogonal transformation to spread out large values before quantization. The paper calls the latter technique incoherent processing; both aim to reduce quantization error in the lower-precision matrix work. Lower storage precision does not imply every normalization accumulator uses FP8. Compare outlier-heavy inputs and final outputs against a higher-precision reference; raw throughput is not evidence of acceptable model quality. Numerics: FP32, BF16, FP8 and FP4 supplies the vocabulary for separating storage, accumulation and output precision.
The FA3 paper reports up to roughly 740 TFLOP/s in FP16 and close to 1.2 PFLOP/s in FP8 on its experiments. These are reported results under the paper's settings, not measurements of this site or promised rates for arbitrary heads, masks and context lengths. Its numerical comparisons also concern specific inputs and baselines; they do not certify every model's tolerance.
Make a comparison another engineer could reproduce
For two candidate kernels, record the GPU variant, software versions, dtype and accumulation mode, batch/head counts, head dimension, query/key lengths, mask, layout, dropout, and whether the measurement covers forward, backward or both. Warm compilation separately and use device timing with synchronization. Check maximum and typical numerical error against the same reference before comparing time.
Use the same FLOP convention. For example, count just the two dense non-causal forward matrix products:
Batch = 1; heads = 2; N = 1,024; d = 64
QK work = 2 × batch × heads × N² × d = 268,435,456 FLOPs
PV work = same count = 268,435,456 FLOPs
Total matrix work = 536,870,912 FLOPs
Check: two equal matrix products double the QK count.
This excludes softmax and other scalar work. The general matrix-work count is 4 × batch × heads × N² × d. Causal work, unequal query/key lengths and backward alter the count. A reported TFLOP/s number can change because the numerator changed, even if time did not. Latency for the actual workload is the final comparison.
Choose a Hopper-specific path when the device supports it, the required semantics are implemented and it wins the representative benchmark. Reverse the choice for unsupported features, accuracy failure or a short/skinny shape where overhead erases the gain. An cannot execute Hopper-only instructions, but that alone does not establish which supported alternative is best. A large speed difference also does not prove which backend ran: inspect the selected kernel and dispatch behavior with Nsight Systems or the framework profiler.
What interviewers probe next
- What did v2 change after IO fusion? More useful parallel work and less non-matmul/coordination overhead.
- What can v3 overlap? Work from different tiles with compatible dependencies; not a softmax with the unavailable scores it consumes.
- Why not always choose FP8? Support and numerical error are part of the contract, along with speed.
Common mistakes
- Describing all three versions only as the same memory optimization.
- Treating a paper's best forward result as an end-to-end training or serving guarantee.
- Removing waits because an instruction is asynchronous, or identifying a backend from speed alone.
Key takeaways
- Versions 1, 2 and 3 address intermediate traffic, work partitioning, then overlap and lower precision.
- Data-ready and buffer-reuse synchronization protect different dependencies.
- Compare matched workloads and numerical tolerances; historical paper peaks are reference results, not deployment forecasts.
Primary sources: FlashAttention, FlashAttention-2, and FlashAttention-3.