AI Infra Interviews logo
← CUDA, Triton & Kernel Engineering / 17
hardTogether AIvLLM

Sketch a paged attention kernel. What changes from FlashAttention once the KV cache is not contiguous?

Follow a logical token through the block table into a physical KV page. Then combine split attention states correctly, including empty splits, and count lookup traffic at the layer and head the kernel actually processes.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Follow a logical token through the block table into a physical KV page. Then combine split attention states correctly, including empty splits, and count lookup traffic at the layer and head the kernel actually processes.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Define the KV layout and query-to-KV-head mapping, preserve stable softmax state across page boundaries, and choose a split count from measured work distribution rather than assumed occupancy.

DISCUSSION · 0

No comments yet — be the first to share your approach.