Design a kernel benchmarking setup that will not lie to you. What does it control for, and how do fake speedups get published?

Six things decide whether a kernel measurement means anything, and a script that ignores them can report a number that is wrong by a factor of two in either direction. What to control, which statistic to report, and the six ways a large speedup turns out to be a measurement artifact.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Foundational

🧮 Open Weights & Serving Engines

Serving Benchmarks That Do Not LieMost published serving numbers are not comparable to each other and not predictive of production, because they differ in the input distribution, the concurrency, whether the cache was warm, and which of several very different metrics is being reported. A benchmark that supports a decision has to fix all four, report a distribution rather than a mean, and be run against the traffic shape you actually serve. The single most useful discipline is to compute the bandwidth bound first, so you know what fraction of the possible you achieved.

Foundational

🖧 Hardware & Cluster Build-Out

Burn-In and Acceptance TestingNew hardware fails early or it fails late, and burn-in exists to move the early failures before the cluster is handed over rather than after. A proper acceptance test runs every layer under sustained load for days, compares every node against its siblings rather than against a specification, and produces a signed number the buyer and the vendor both agree on. The comparison is the important part: identical hardware running identical work should produce identical numbers, and the outliers are the finding.

Foundational

⚡ Kernels & Compilers

CUDA Programming ModelCUDA splits a program into a host that allocates, copies and enqueues work, and a device that runs thousands of identical threads organized as a grid of blocks. Getting the split right, and knowing that a launch returns before the kernel runs, decides whether your first live-coding kernel produces a correct number or a silent zero.

Core

⚡ Kernels & CompilersSign in

Memory CoalescingA warp's 32 threads issue one memory request together, and the hardware serves it in 32-byte sectors. Coalescing is arranging addresses so those sectors are full of bytes the warp will use. It decides whether a bandwidth-bound kernel moves at the HBM rate or at an eighth of it, and it is the pattern NVIDIA's trace-classification interview question tests.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour fused attention kernel matches the reference at 512 tokens and drifts at 8,000. Find the bug.Closest to this topicDesign the benchmark you would run before putting a new open-weights model into production.Closest to this topicTell me about a performance win you are proud of. How did you measure it?

FEDITOR'S NOTE

Scored on naming warmup, clocks, cache state, timing method and statistics as separate controls, on reporting against a hardware ceiling rather than against a baseline, and on recognising the artifact patterns.

DISCUSSION · 0

No comments yet — be the first to share your approach.