AI Infra Interviews logo

Write a fused RMSNorm kernel in Triton and benchmark fair baselines

Implement masked RMSNorm, check three dtypes and compare eager PyTorch, torch.compile, native RMSNorm and a custom GPU kernel.

PremiumAdvanced GPU project

AI Infra Interviews editorial · Updated

How do you show that a fused RMSNorm kernel is useful?

First validate the formula, masking, shape and dtype against a reference, including widths that are not powers of two. Then compare warmed eager, compiled and native implementations using the same inputs and timing procedure. A custom kernel beating eager tensor operations does not prove an advantage over an optimized baseline or improve an entire model by the same ratio.

Read the primary sources
What you build
A forward-only kernel, numerical checks, repeated timing records and an optional profiler trace.
Environment
Linux · CUDA NVIDIA GPU · executed on RTX 5070 12 GB
Plan your session
2–4 hours plus CUDA environment setup. A planning estimate, not measured runtime.
Before you begin
Read tensor shapes, reductions and Python functions. Use a separate PyTorch/Triton CUDA environment; this project does not require a serving cluster.
GPU kernel and four baselines executed. Executed on RTX 5070 with PyTorch 2.14.1+cu130 and Triton 3.8.0: 54 numerical cases, invalid-input checks and three timing blocks per shape. Compiled and native baselines can be competitive; timings do not establish whole-model speedup.
  1. 01Load a row

    Mask padded lanes and accumulate squared values in FP32.

  2. 02Normalize

    Divide by the true row width, add epsilon and multiply by the learned weight.

  3. 03Check then time

    Reject numerical errors before comparing four warmed implementations.

Follow the changing state through the project. Each boundary has a check in the walkthrough.

Continue with Premium

Premium includes this guide’s complete walkthrough and downloadable project files. The technical overview, prerequisites and verification notes above are public.

View Premium access

Start with Train a tiny language model from scratch with PyTorch (Free to read).

Primary sources

Checked . Source review and execution checks are described separately above.