cuBLAS already gives you a fast GEMM. What does CUTLASS give you that it does not, and when is that worth it?

A library call gives you a tuned matmul and nothing around it. Four things a template library gives instead: an epilogue fused into the output stage, one launch for a list of ragged problems, a matmul whose inputs are different types, and a tile schedule that fills the SMs on awkward shapes.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

⚡ Kernels & Compilers🔒 Premium

CUTLASS and Tensor Core KernelsCUTLASS is NVIDIA's template library for building GEMM-shaped kernels that run tensor cores at near cuBLAS speed while letting you change the data types, the tile shapes and the epilogue. Its hierarchy (device, kernel, collective mainloop, tile, instruction) is the vocabulary of every tensor-core discussion, and knowing when it beats calling cuBLAS or writing Triton is the judgment question kernel interviews end on.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsPlanning a fleet for a sparse open-weights model works differently from planning one for a dense model, because memory follows total parameters and throughput follows active parameters, and those now differ by more than twenty times. The sizing goes in one direction only: from a traffic forecast to tokens per second, to replicas at a measured operating point, to GPUs, to racks and kilowatts. Doing it in the other direction, from an available GPU count, produces a fleet that fits the hardware rather than the demand.

Foundational

🧮 Open Weights & Serving Engines

Reading config.json to Size a Model You Have Never RunEvery Hugging Face model ships a config.json, and it contains enough to compute the weight footprint, the KV cache per token, the parallel degrees that divide cleanly and the minimum GPU count, before downloading a byte. Doing that derivation is a standard whiteboard exercise in serving interviews because it is exactly what an engineer does on the morning a new model lands, and the fields that matter are the same across every recent architecture.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEA mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA take-home gives you a working layernorm kernel at a tenth of memory bandwidth. Make it fast and justify every change.Next in this trackImplement the verification step of speculative decoding, including the rollback of the KV cache after a rejection.Next in this trackHow do you overlap host-to-device transfers with compute, and what has to be true before the overlap actually happens?

FEDITOR'S NOTE

Scored on naming the four capabilities with the situation each answers, on quantifying the epilogue and tail-wave savings, and on saying honestly where cuBLAS remains the correct choice.

DISCUSSION · 0

No comments yet — be the first to share your approach.