Roofline Model

TL;DR: Attainable FLOPS = min(peak FLOPS, arithmetic intensity × memory bandwidth). The knee of that curve is the ridge point, peak ÷ bandwidth: 989 TFLOPS ÷ 3.35 TB/s ≈ 295 FLOP per byte on an H100 in bf16. Decode at batch 1 has an intensity of about 1 and attains about 3.4 TFLOPS; a large GEMM sits in the thousands and attains the peak. Raising intensity (fusion, batching, lower-precision weights) is the only way up the slope; more threads do nothing once you are on it.

The one picture

Every kernel does some arithmetic and moves some bytes to do it. Divide the FLOPs by the bytes and you get its arithmetic intensity, in FLOP per byte. Plot intensity on a log x axis and attainable throughput on a log y axis, and the hardware draws a roof over the chart: a sloped line where throughput = intensity × bandwidth, and a flat line at peak FLOPS. A kernel lands under whichever is lower.

The ridge point is where the two lines meet, and it is a property of the hardware alone: peak FLOPS divided by memory bandwidth. Using the dense bf16 numbers from the spec sheet:

Read the table from the H100 to the H200: same compute, more bandwidth, so the ridge moves left and a memory-bound kernel gets faster without changing a line. That is the entire argument for the H200 in decode-heavy serving. The full spec sheet lists the rest, and the roofline calculator draws this chart for any part and any intensity.

Where the common kernels sit

Intensity is a property of the operation, so you can place the kernels an LLM runs before profiling anything.

A decode step for a dense model reads every weight once and does two FLOPs per weight per sequence (a multiply and an add). With bf16 weights at 2 bytes each, that is 2 × batch FLOPs per 2 bytes: intensity ≈ batch. At batch 1 the point sits near 1 FLOP per byte, and attainable throughput is 1 × 3.35 TB/s ≈ 3.4 TFLOPS on an H100, about 0.3% of peak. This is not a badly written kernel; it is the physics of reading 141 GB of weights to produce one token. Batch 64 moves the point to 64 and attains about 214 TFLOPS. Only past the ridge, near batch 300, does the step become compute-bound, which is why throughput per GPU keeps improving with batch until roughly there and then stops.

Prefill is the opposite. A GEMM over a 2,048-token prompt reuses each weight 2,048 times, so intensity is in the thousands, far past the ridge, and the tensor cores are the limit. FlashAttention lands in between: by tiling the score matrix in shared memory it reaches an intensity around 100 to 200 on long sequences, where the naive attention it replaced sat well below 10 because it wrote the N² matrix out to HBM and read it back.

Elementwise operations are the floor. A LayerNorm or a GELU reads each element, does a handful of FLOPs, and writes it back: intensity under 1. Nothing you do inside the kernel changes that; the fix is to stop the round trip, which is what kernel fusion means.

Using it, not just drawing it

The roofline answers the first question a performance engineer asks about any kernel: which wall am I against? The procedure is three numbers. Get FLOPs from the operation (2 × M × N × K for a GEMM). Get bytes from the tensors actually moved (weights read once, activations read and written). Divide, compare with the ridge. Nsight Compute prints achieved DRAM throughput and achieved FLOPS directly, and its "roofline" section plots the point for you; the manual version is what you do on the whiteboard.

Then the levers follow from the position. Under the roof on the slope: reduce bytes (fusion, lower-precision weights, a smaller KV cache) or raise reuse (batching, tiling). On the flat roof: better tensor-core utilization, better tile shapes, fewer wasted lanes. A kernel at 30% of peak that is on the slope cannot be fixed by "using the tensor cores harder", and a kernel on the roof cannot be fixed by fusion. Most performance mistakes in interviews are applying the wrong lever to the wrong regime.

One refinement that earns marks: the roofline has more than one slope. Shared memory and L2 have their own bandwidths, far above HBM's, so a kernel can be HBM-bound while sitting under the L2 roof, or compute-bound against a lower ceiling because it uses fp32 CUDA cores rather than bf16 tensor cores. Naming which roof applies (HBM, L2, tensor core peak at the precision in use) is the difference between a textbook answer and an engineer's.

What interviewers are listening for

"Is decode memory-bound or compute-bound, and how do you know?" is asked at nearly every serving company. The scored answer names intensity ≈ batch for bf16 weights, the ridge around 295 for an H100, and the consequence: single-stream decode is hopelessly memory-bound and batching is the lever. The follow-up is "so what happens with fp8 weights?", and the answer is that bytes halve, intensity doubles at the same batch, and the compute peak also doubles, so the ridge stays roughly where it was while the kernel moves right.

The trap to avoid is quoting the sparse peak. NVIDIA's headline 1,979 TFLOPS for H100 bf16 includes 2:1 structured sparsity that dense GEMMs do not have; use 989 and say why. An interviewer at a chip company will notice.

Key takeaways

• Attainable = min(peak, intensity × bandwidth). The ridge point, peak ÷ bandwidth, is about 295 FLOP/B on an H100 in bf16.
• Decode intensity ≈ batch (bf16 weights): batch 1 attains ~3.4 TFLOPS, and only near batch 300 does decode become compute-bound.
• On the slope, reduce bytes or raise reuse; on the roof, use the tensor cores better. Applying the wrong lever is the classic mistake.
• Elementwise kernels live under 1 FLOP/B; fusion is the fix, not optimization inside the kernel.
• Use dense peaks. The sparse marketing figure doubles the roof and no dense kernel reaches it.