TL;DR: Arithmetic intensity is floating-point work divided by bytes transferred across a specified memory boundary. The roofline bounds throughput by
min(P, I × B). For SXM dense BF16 tensor cores and HBM, the ridge is about 295 FLOP/byte. Use it to identify the tighter modeled ceiling, then compare measured performance before diagnosing the kernel.
How to approach it
Define work, byte count, compute path and memory boundary. Derive the ridge, then count one matrix multiplication completely. Show both a throughput ceiling and a time floor. Finish by explaining what a measurement below the ceiling would make you test, including when that proposed change would lose.
A strong answer
A GPU can multiply values only after it has data to work on. A matrix product with few rows may move almost the same weight matrix as one with many rows, but do much less arithmetic with each loaded weight. That gives the two products different ceilings even on the same device.
The Roofline Model expresses this with I = F/D: floating-point operations divided by bytes. Let P be the applicable compute peak and B the bandwidth across the boundary counted in D.
R ≤ min(P, I × B) throughput ceiling
T ≥ max(F/P, D/B) time floor
ridge = P/B
H100 SXM, dense BF16 tensor-core arithmetic and HBM:
P ≈ 989 × 10^12 FLOP/s; B = 3.35 × 10^12 byte/s
ridge ≈ 989 / 3.35 = 295 FLOP/byte
The H100 specification marks its headline BF16 figure as sparse; use the dense value for dense work. A different precision, instruction path or memory boundary changes the selected roof. About 295 is not the right ridge for every kernel on an H100.
Now multiply A[M,K] × B[K,N] to produce C[M,N], with K = N = 8,192. Assume two-byte BF16 inputs and output, FP32 accumulation in the tensor-core computation, no read of an old C, and no extra epilogue or workspace traffic. The following is an ideal cold-data traffic model: read each input once from HBM and write the output once.
F = 2MNK count a multiply-add as two FLOPs
D = 2(MK + KN + MN) bytes two inputs and one output
I = F/D
M = 128:
F = 17,179,869,184 FLOP
D = 138,412,032 bytes = 132 MiB
I ≈ 124.12 FLOP/byte
F/P ≈ 17.37 µs; D/B ≈ 41.32 µs
T ≥ 41.32 µs; R ≤ 415.81 TFLOP/s
M = 8,192:
F = 1,099,511,627,776 FLOP
D = 402,653,184 bytes = 384 MiB
I ≈ 2,730.67 FLOP/byte
F/P ≈ 1,111.74 µs; D/B ≈ 120.19 µs
T ≥ 1,111.74 µs; R ≤ 989 TFLOP/s
Sanity check: increasing M by 64 multiplies work by 64, but ideal traffic grows only from 132 to 384 MiB because the large B matrix is unchanged. MiB uses 2^20 bytes; the specified TB/s uses 10^12 bytes per second. Keeping those units distinct avoids a quiet timing error. The matrix-multiplication guide develops this operation and byte count.
Both floors constrain the same execution. The kernel cannot finish before it has done the counted arithmetic or moved the counted bytes. Taking the larger floor allows those activities to overlap; adding them would assume they happen one after the other. If a measurement is slower than that floor, investigate the gap before choosing an optimization.
For 128 rows, the HBM estimate gives the larger floor. For 8,192 rows, the compute estimate does. Neither calculation is a benchmark. Cache state, repeated loads, tile scheduling and instruction overhead can change the real execution. If inputs were already cached, the cold-data byte count would not describe measured HBM traffic; if the implementation reloads them, it may undercount. The GPU Memory Hierarchy is why those distinctions matter.
Suppose, as an illustrative result, the 128-row kernel takes 80 µs. Its useful-work throughput is 17,179,869,184 / 80 µs ≈ 214.75 TFLOP/s. If the measured HBM byte count also matched the ideal 132 MiB, its byte rate would be about 1.73 TB/s, around 52% of the stated peak. That gap does not identify a single cause. In particular, this isolated matrix product has no KV-cache reads to blame.
I would compare actual traffic with the estimate and inspect instruction issue, synchronization and available parallel work using the matching Nsight Compute roofline and memory views. If repeated HBM loads dominate, test better reuse. If traffic is already low and execution stalls elsewhere, test that cause first. A layout or tiling change loses its case if extra synchronization or register pressure increases total time.
More rows can amortize weight reads, but this example does not promise a 64× serving-throughput gain. A serving step includes other operations, activation and attention traffic, and request-latency constraints. Similarly, fusion can save intermediate transfers while introducing other costs. Recompute the changed work and traffic, then measure the required output.
What interviewers probe next
- “What if the data comes from L2?” Count L2 bytes and use its bandwidth for an L2 roof. HBM intensity uses HBM bytes, which can be much smaller on a warm execution. Never mix one level's traffic with another level's bandwidth.
- “Do FP8 weights double performance?” They can reduce weight bytes, but weight-only storage does not prove FP8 arithmetic. Dequantization, other traffic and the actual instruction path decide which compute roof applies. There is no automatic 2× speedup.
- “How does model FLOPs utilization relate?” Divide the declared model FLOPs by elapsed step time and the matching hardware peak, including the device count. State whether recomputation is excluded. An unweighted average of kernel percentages does not give model FLOPs utilization.
- “Can I raise intensity by doing extra FLOPs?” Numerically yes, if you count them. That does not improve time for the useful result. Keep useful-work and executed-operation counts separate when comparing kernels.
Common mistakes
- Forgetting an input or output in the traffic count, or using a cold-data estimate as a measured HBM count.
- Treating a point left of the ridge as proof that bandwidth is already saturated.
- Calling a time floor an achieved latency, or applying a sparse/tensor-core peak to the wrong computation.
- Turning weight reuse or lower precision into an unconditional application speedup.
Key takeaways
- The stated H100 dense-BF16/HBM ridge is about 295 FLOP/byte; name those assumptions.
- At 128 rows, the ideal example gives a 41.32 µs memory floor; at 8,192 rows, a 1.112 ms compute floor.
- Compare measurements with a consistent work and traffic definition before assigning the bottleneck.
- A change succeeds when the required result arrives faster, not merely when a plotted point moves right.