attention
AI infra interview questions tagged attention, across every topic.
7 questions · 4 unlocked for you
Concepts behind "attention"
The curriculum that explains the ideas these questions test.
Foundational
The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.🚀 Inference & Serving
Foundational
Multi-Head Latent Attention and Sparse IndexersGrouped-query attention shrank the KV cache by sharing key and value heads. Latent attention goes further by caching a single compressed vector per token per layer and reconstructing the heads on the fly, which cuts the cache by tens of times rather than by a small factor. On top of that, sparse indexers pick a few thousand relevant positions per query instead of attending to all of them, turning the quadratic term linear at long context. Both are now standard in open-weights models, and both change how a serving deployment is sized.🧮 Open Weights & Serving Engines
Core
Attention Variants: MHA, GQA, MQA and MLAThe KV cache scales with the number of key-value heads, and the four attention variants differ exactly there: multi-head keeps one KV head per query head, multi-query keeps one for all, grouped-query shares one across a group, and multi-head latent attention caches a compressed latent instead of keys and values at all. For Llama 3.1 70B that is the difference between 2.6 MB and 320 KB per token; for DeepSeek-V3 it is about 70 KB. The variant a model was trained with is a serving decision made before the first GPU was bought.🚀 Inference & ServingSign in
Core
Arithmetic Intensity by OperationThe roofline says a kernel's ceiling is set by its FLOPs per byte against the hardware's ridge point. This page does the FLOPs-per-byte arithmetic for the operations an LLM actually runs (decode at several batch sizes, prefill, the attention score matmul with and without FlashAttention, LayerNorm, an embedding lookup) so the reader can place any of them on the roofline from first principles and say which lever moves it. The numbers explain why a serving fleet's GPUs report 30% utilization while fully loaded.🧮 Napkin Math & CapacitySign in
Advanced
FlashAttention InternalsStandard attention writes the N x N score matrix to HBM and reads it back, which makes it memory-bound and quadratic in memory. FlashAttention tiles Q, K and V through shared memory, keeps a running max and sum so the softmax never needs the full row, and recomputes scores in the backward pass. Knowing the online-softmax rescale, why FlashAttention-2 flipped the loop order, and what FlashAttention-3 overlaps on Hopper is the difference between naming the paper and being able to write the kernel.⚡ Kernels & Compilers🔒 Premium