mediumNewNVIDIAGoogle

Write a tiled matrix multiply and explain the cache effect. Then say honestly what your benchmark actually measured.

The tiling is ten lines and the theory is a working-set calculation. The interesting part is that the obvious benchmark does not measure what it appears to, because the library underneath is already tiled, and knowing which effect your numbers contain is the skill being tested.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

Foundational

💻 Coding for Infra

Cache-Friendly Data StructuresA cache line is 64 bytes and it is the unit of coherence, so where data sits decides how fast code runs more often than which algorithm it uses. Two consequences dominate infrastructure code: a lookup that chases a pointer pays two dependent memory stalls instead of one, and two threads updating adjacent variables contend for a line they do not logically share. Both are layout problems with layout fixes.

Advanced

⚡ Kernels & Compilers🔒 Premium

Tiled Matrix MultiplicationA matrix multiply has enough reuse to be compute-bound, but only if the kernel captures that reuse in shared memory and registers instead of re-reading HBM. Tiling is how: a block owns an output tile, streams K-slices of A and B through shared memory, and each thread accumulates a small register tile. It is the live-coding exercise that separates people who know the roofline from people who have climbed it.

Advanced

⚡ Kernels & Compilers🔒 Premium

FlashAttention InternalsStandard attention writes the N x N score matrix to HBM and reads it back, which makes it memory-bound and quadratic in memory. FlashAttention tiles Q, K and V through shared memory, keeps a running max and sum so the softmax never needs the full row, and recomputes scores in the backward pass. Knowing the online-softmax rescale, why FlashAttention-2 flipped the loop order, and what FlashAttention-3 overlaps on Hopper is the difference between naming the paper and being able to write the kernel.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackImplement a log-structured event store for fleet metrics with range queries and compaction.Next in this trackA multi-GPU training job hangs at step 400 with every GPU at 100 percent utilization. Debug it.Closest to this topicDesign the benchmark you would run before putting a new open-weights model into production.

FEDITOR'S NOTE

Scored on the working-set arithmetic and the reuse factor, on writing correct tiled code, and on recognising that a numpy benchmark measures dispatch overhead rather than cache locality because BLAS is already blocked.

DISCUSSION · 0

No comments yet — be the first to share your approach.