Kernels and the Machine
Reason from a warp to the profiler: how the machine schedules work, why access pattern is most of memory performance, how a reduction, a tiled matmul and a fused kernel are built, and how to take a slow kernel to the roofline with evidence a reviewer can check.
A kernel is fast for a reason you can name, or it is fast by accident and will not stay that way. This course is about naming the reason. It starts with the unit the hardware actually schedules, moves to the memory system that sets most kernels' speed limit, builds the three kernel shapes every fast kernel borrows from, and ends with the evidence: a profile read correctly and an analysis written so that a reviewer can check every claim.
Four modules carry that argument. The execution model covers warps, occupancy and launch overhead. The memory hierarchy covers coalescing, shared memory and reuse. The kernel module writes a reduction, a tiled matmul and a fusion, and accounts for what each step buys. The evidence module turns the roofline into a method, reads an IO-aware attention kernel as a traffic argument, and closes with the capstone: take a slow kernel to the roof and write the analysis.
Hardware changes. The lessons are written in symbols, and the numbers for the current generation live in one dated box per lesson, so the course stays true when the next chip ships.
When you finish, you can
- Cost a kernel in warp instructions and stalls rather than in threads
- Decide when occupancy is the right knob and when raising it makes a kernel slower
- Identify launch overhead in a timeline and choose between fusion, graphs and batching
- Predict memory traffic from an access pattern before running anything
- Use shared memory as a managed cache and detect bank conflicts from the arithmetic
- Explain why reuse, and only reuse, moves a kernel up the roofline
- Write a reduction and a tiled matmul, and account for each speedup
- Read a profile, name the binding resource, and write the analysis a reviewer can check
BEFORE YOU START · Inference Engineering or Training at Scale, and comfort reading C++. This is the deepest course and the narrowest. It assumes the roofline and the memory budget from Foundations and spends its time inside a single kernel.
Syllabus
The execution model
Cost a kernel in the units the machine schedules: warp instructions, resident warps and launches. Count the tail, price a divergent branch, decide when occupancy is the wrong knob, and find the launch overhead that dominates small-batch steps.
The memory hierarchy
Predict a kernel's traffic from its access pattern before running it, decide when staging through shared memory pays and check the bank arithmetic at design time, and see why reuse at every level of the hierarchy is the only lever that moves a kernel toward the roofline's ridge.
The kernel
Write the three kernel shapes every fast kernel borrows from: a reduction whose three levels each exist for a reason, a tiled matmul read as six parts with two barriers and an epilogue, and a fusion that deletes the traffic between operations that no tiling can touch. Each comes with the account of why the naive version is slow.
The evidence
Turn the roofline into a seven-step method a reviewer can check, read IO-aware attention as the reuse law's hardest case rather than a faster matmul, and finish with the capstone: take a slow kernel to the roof, decide whether it should exist at all, and write the analysis memo.