Kernels and the Machine
Reason from a warp to the profiler: how the machine schedules work, how access patterns and cache behavior affect memory performance, how a reduction, a tiled matmul and a fused kernel are built, and how to take a slow kernel to the roofline with evidence a reviewer can check.
A change to lane mapping can reduce cache transactions while leaving device-memory traffic unchanged, so a count at one level does not tell you the count at the next. This course teaches you to follow the work from the lanes through the caches to device memory, calculate a useful bound, and choose the next experiment from evidence.
Seven modules build that skill. The execution model covers warps, occupancy and launch overhead. The memory hierarchy connects coalescing, shared memory and reuse to traffic. The kernel module develops reductions, tiled matrix multiplication and fusion. The evidence module combines them: reconstruct online attention, interpret a profile, and compare a complete operation before choosing an optimization. The compiled-machine module then moves to an accelerator whose compiler owns the kernel, using TPUs as the example: shapes become programs, layouts become communication, and training and serving each need their own discipline.
The work-mapping module then connects CUDA compilation artifacts, coordinate layouts and grid scheduling. The correctness module develops scan-based output ownership and an adversarial evaluator before a GPU qualification project.
You finish with a performance memo that separates assumptions, calculations and measurements. It includes correctness checks, a predicted change, the measured result when hardware is available, and the conditions that would reverse the recommendation. The earlier roofline capstone is a synthetic analysis exercise. The final capstone links to a GPU project with executed correctness checks and comparisons against eager and compiled baselines. A separate CPU project makes the work-mapping calculations executable without a GPU.
Mechanisms and equations form the core of each lesson. Product-specific behavior and hardware examples live in dated notes so those references can be checked as the tools change.
When you finish, you can
- Cost a kernel in warp instructions and stalls rather than in threads
- Decide when occupancy is the right knob and when raising it makes a kernel slower
- Identify launch overhead in a timeline and choose between fusion, graphs and batching
- Calculate traffic bounds and distinguish cache-sector requests from measured device-memory traffic
- Use shared memory as a managed cache and detect bank conflicts from the arithmetic
- Predict how reuse, representation and algorithm changes alter arithmetic intensity
- Write a reduction and a tiled matmul, and account for each speedup
- Read a profile, name the binding resource, and write the analysis a reviewer can check
- Carry the method to a compiler-scheduled accelerator: buckets, sharding layouts, a ported training step and a serving shape plan
- Trace a source change to PTX and native artifacts, and separate compilation from steady execution
- Map coordinates and balance tiled work while preserving reduction and buffer ownership
- Qualify a kernel with adversarial cases and defend a measured choice against a compiled baseline
BEFORE YOU START · Inference Engineering or Training at Scale, and comfort reading C++. This is the deepest course and the narrowest. It assumes the roofline and the memory budget from Foundations and spends its time inside a single kernel.
Put the course into practice
Build a project after the linked lesson. Keep the scripts, measurements and written decision as evidence of your work. Hardware requirements and execution limits appear on each project.
- Premium · Advanced GPU project
Write a fused RMSNorm kernel in Triton and benchmark fair baselines →
A forward-only kernel, numerical checks, repeated timing records and an optional profiler trace.
- Premium · Advanced
Model GPU work mapping and parallel scans →
Executable work-mapping models, a synthetic evidence file and tested ownership calculations.
- Premium · Advanced
Qualify and benchmark a Triton kernel →
A correctness-qualified experiment with raw matched timings, compiled artifacts and a scoped implementation decision.
Syllabus
The execution model
Cost a kernel in the units the machine schedules: warp instructions, resident warps and launches. Count the tail, price a divergent branch, decide when occupancy is the wrong knob, and find the launch overhead that dominates small-batch steps.
The memory hierarchy
Predict a kernel's traffic from its access pattern before running it, decide when staging through shared memory pays and check the bank arithmetic at design time, and see why reuse at every level of the hierarchy is the only lever that moves a kernel toward the roofline's ridge.
The kernel
Write the three kernel shapes every fast kernel borrows from: a reduction whose three levels each exist for a reason, a tiled matmul read as six parts with two barriers and an epilogue, and a fusion that deletes the traffic between operations that no tiling can touch. Each comes with the account of why the naive version is slow.
The evidence
Use matched traffic boundaries and controlled experiments to explain a profile. Derive tiled attention and merge partial softmax state, then compare complete-operation timings and reuse in a performance memo.
The compiled machine
Carry the course's methods to an accelerator where the compiler owns the kernel: shapes become programs, layouts become communication, and training and serving each need their own discipline to stay fast and correct.
The work mapping
Connect compiled artifacts, tensor layouts and grid scheduling to one inspectable performance explanation.
The correctness case
Build safe offsets, qualify a kernel contract and defend a measured replacement against a compiled baseline.