hardNVIDIA

What can you rely on from the L2 cache on a GPU? When does it save a kernel, and when does it mislead you?

Calculate the data a kernel could reuse, then test whether it stays in L2. Distinguish capacity, cache-policy hints and measured traffic from a guarantee of residency.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Calculate the data a kernel could reuse, then test whether it stays in L2. Distinguish capacity, cache-policy hints and measured traffic from a guarantee of residency.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational

🧩 GPU & Accelerator Architecture

GPU Memory HierarchyTrace storage scope and data reuse from registers through shared memory, caches and HBM. Worked register and traffic examples explain why a source-level load is not automatically an HBM transfer.

Advanced

⚡ Kernels & Compilers🔒 Premium

Tiled Matrix MultiplicationTiled matrix multiplication reuses input values across many output calculations. Shared-memory and register tiles reduce repeated data movement while consuming limited on-chip resources.

Advanced

⚡ Kernels & Compilers🔒 Premium

Shared Memory and Bank ConflictsShared memory lets threads in a block exchange and reuse data. Bank conflicts occur when an access needs multiple distinct words from the same bank.

Advanced

⚡ Kernels & Compilers🔒 Premium

FlashAttention InternalsFlashAttention computes exact attention in tiles without storing the full score or probability matrix in device memory. Follow the running softmax state, understand why inputs can still be reread, and distinguish the mechanisms introduced in the first three papers.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat does ECC on a GPU cost you, and why do you keep it on across a fleet?Next in this trackExplain NUMA and CPU affinity on a GPU node. Why can the wrong socket halve host-to-device bandwidth?Next in this trackCompare A100, H100 and B200 for a 70B serving fleet. Which gives the most tokens per dollar, and where do fp8 and fp4 change the ranking?

FEDITOR'S NOTE

Explain reuse distance and competing traffic before predicting cache hits. Persistence controls influence eviction; they do not pin arbitrary data or establish a universal usable-capacity threshold.

DISCUSSION · 0

No comments yet — be the first to share your approach.