OpenAI CUDA, Triton & Kernel Engineering interview questions

A chain of five elementwise operations reads and writes the tensor five times when once would do. The byte arithmetic for an unfused chain against a fused one, the launch overhead that matters at small sizes, the three fusion shapes, and the operations where fusion changes nothing because the matmul was at the roof.

One program per row, the row in registers, max then exp then sum then divide, one read and one write of HBM. The runnable kernel with its launch, the numerics (subtract the max, accumulate in fp32), the block-size rule and the wide-row limit, and the arithmetic that says the kernel is done at 90% of copy speed.

Standard attention writes an N by N score matrix to HBM and reads it back twice. At 8k context that is 134 MB per head. The traffic arithmetic before and after tiling, the online-softmax identity that makes one pass possible, where the kernel lands on the roofline, and why the memory saving matters more than the speed.

Triton hands you a block of data and writes the thread-level code for you, which covers most of what a serving or training stack actually needs. The four things it does not give you, the performance you give up in each case with numbers, the reversal as Triton gains Hopper features, and how to decide in the room.

Four stages: a bytecode interpreter captures a graph with guards, an autograd pass splits forward from backward, a compiler emits Triton for everything but the matmuls, and a mode that replays the step as one launch. What each buys, and the two failures that silently give it back.

A row of logits is half a megabyte, so the kernel is not bandwidth-bound and the cost is in how many passes you make over it. Why a full sort does far more work than the question asks, the two shapes that fit small and large k, and the four-pass radix select with a reference that matches a sort exactly.

Compute the utilization first so you know how much time is unaccounted for, then read one timeline in a fixed order: GPU idle, then what the host was doing in the gaps, then whether the gradient all-reduce overlapped the backward pass, then the optimizer. Four causes, the evidence for each, and what each is worth.

The kernel is four lines; the interview is about the other forty. How a global index comes out of block and thread ids, why the bounds check exists, what the launch configuration means for a 100M-element array, and the arithmetic that shows the host-to-device copies cost 50 times more than the add.

A warp issues one load instruction and the memory system turns it into some number of 32-byte sector requests; that number is the whole story. The arithmetic for contiguous, stride-2 and stride-32 access, the row-major matrix where a loop order change gives 8x, and the two Nsight Compute counters that show the waste.

Thirty-two addresses per warp instruction, thousands of instructions: say what the pattern is and how many sectors it cost. The address-delta test for contiguous and strided, the sector count that measures the damage, the code that does both, and the edge cases (misalignment, inactive lanes, mixed widths).

Shared memory has 32 banks, each 4 bytes wide, and a warp's access is as slow as the most-loaded bank. The bank of an address, why a column walk down a 32-wide tile puts all 32 lanes in one bank, the padding by one column that spreads them across all 32, and the profiler counter that confirms the fix.

The naive kernel reads two bytes per multiply-add; a 128x128 tile with register blocking reads a few hundredths of that. Each step (shared-memory tiles, register blocking, vectorized loads, double buffering, tensor cores) with its intensity arithmetic and speedup, and where a hand kernel stops and cuBLAS begins.

Occupancy is how many warps an SM holds against its maximum, and it is a means, not an end. The arithmetic from registers per thread to resident warps, the launch bound that caps registers and the spills that follow, the profiler's occupancy view, and the kernel where halving occupancy doubled speed.

Each version fixed a different bottleneck: version 1 fixed memory traffic, version 2 fixed non-matmul work and parallelism, version 3 fixed the fact that softmax and matmul were waiting for each other on Hopper. The arithmetic that shows why a softmax costing one percent of the FLOPs can cost sixty percent of the time.

Four sections, read in a fixed order, and one number that usually names the bottleneck outright. The report of a kernel at 12 percent of DRAM bandwidth while its memory pipeline reads 82 percent busy, what that gap means, the fix it implies, and the numbers the fixed kernel reports back.

A sum reads every byte once and does one add per element, so the only question is whether you reach the bandwidth ceiling. The ceiling in milliseconds, four versions from an atomic per element to warp shuffles with sixteen loads in flight, and the bytes-in-flight math behind the last jump.

A transpose moves every byte exactly once, so a device copy is the honest ceiling. Why the naive kernel pays eight times the write traffic, why staging through shared memory does nothing until you pad, and the counter that tells the two problems apart.

At batch 1 a decode step is a few milliseconds of GPU work behind several hundred kernel launches, and the CPU cannot issue them fast enough. The arithmetic showing the GPU starve, what replay changes, and the four constraints capture imposes on memory.

The KV for one sequence is scattered across fixed-size pages, so the kernel reads a table of physical block numbers before it can read any keys. What the indirection costs, what the table costs in memory, why decode must split the KV dimension across thread blocks, and how the partial softmax states combine.

A library call gives you a tuned matmul and nothing around it. Four things a template library gives instead: an epilogue fused into the output stage, one launch for a list of ragged problems, a matmul whose inputs are different types, and a tile schedule that fills the SMs on awkward shapes.

Normalization reads a row and writes a row, so a copy sets the ceiling and everything else is overhead you can remove. The six changes in the order a reviewer wants them, what each is worth, and the one that is a correctness fix rather than a speed fix, with the measured error that proves it.

The target runs one forward pass over all k drafted tokens, compares its distribution against the draft's at each position, and keeps the longest prefix that survives a coin flip. The acceptance rule that makes the output exactly a target sample, the resampling at the first rejection, and the one-integer rollback.