TL;DR: Tile GEMM so each loaded operand serves multiple outputs, keep the output accumulators alive across the entire K loop, then map the work to suitable matrix instructions. Count reuse separately at global memory, shared memory and registers. “70% of peak” is meaningful only with a matching dtype, dense/sparse convention, shape and measured time; it is not a universal success threshold.
How to approach it
Start with general matrix multiplication (GEMM): identify who writes each output and how its sum accumulates across the K dimension. Add block reuse, then thread or instruction-level reuse. Explain synchronization and boundary handling before discussing pipelining. Finish with a measured bottleneck and a change you would undo if it harms runtime or accuracy.
A strong answer
A tiled multiplication works when K fits in one tile but gives the wrong result for two tiles. Before changing tile sizes, inspect the accumulator's lifetime. Initializing it inside the K loop discards earlier partial sums, even when every load is coalesced.
Three levels of reuse
For C = A @ B, with shapes M×K and K×N, a simple thread computes one output through K multiply-adds. Counting a multiply-add as two FLOPs, loading two FP32 operands per operation gives 2/8 = 0.25 FLOP per requested byte. BF16 operands instead give 2/4 = 0.5. These are source-level operand requests, not guaranteed HBM traffic: broadcasts and caches may already reuse them. For row-major B, neighboring output columns can access neighboring B elements; naive GEMM is not inherently uncoalesced in every mapping.
A block producing a BM×BN output tile stages BM×BK elements from A and BK×BN from B for each K slice. With operand size s bytes, it performs 2 × BM × BN × BK FLOPs per s × BK × (BM + BN) staged input bytes. This excludes output traffic and reuse between blocks. With BM=BN=128, BK=32 and BF16 operands, that is 1,048,576 FLOPs / 16,384 bytes = 64 FLOP/B. FP32 operands reduce that staging ratio to 32.
Within the block, a scalar FP32 thread microtile of 8×8 uses eight A and eight B values for 64 multiply-adds: 128 FLOPs / 64 shared-memory operand bytes = 2 FLOP/B. That is eight times the 0.25 ratio for repeatedly fetching two values per scalar operation at that level. It is not another multiplier to apply blindly to the block's HBM ratio. Tensor Core instructions have their own operand layouts and reuse; Tensor Cores and Matrix Units explains that mapping.
Preserve the reduction and protect the buffers
Initialize each output accumulator before processing any K slice. Cooperatively stage the slice, wait until all required loads complete, consume the slice, and prevent its buffer from being overwritten while another thread still reads it. Repeat, then store each valid output once. Partial edge tiles need masked loads or explicit zero padding and guarded output writes. Threads whose output is outside the matrix may still need to participate in block synchronization.
The executed CPU reference below demonstrates accumulator lifetime and ragged tiles; NumPy handles the inner multiplication. It is not a CUDA kernel or a GPU performance model.
import numpy as np
def tiled_reference(a, b, bm=16, bn=16, bk=8):
if a.ndim != 2 or b.ndim != 2 or a.shape[1] != b.shape[0]:
raise ValueError("incompatible matrices")
if any(type(t) is not int or t <= 0 for t in (bm, bn, bk)):
raise ValueError("tile sizes must be positive integers")
a, b = a.astype(np.float64), b.astype(np.float64)
m, k = a.shape
n = b.shape[1]
out = np.zeros((m, n), dtype=np.float64)
for i in range(0, m, bm):
for j in range(0, n, bn):
acc = np.zeros((min(bm, m-i), min(bn, n-j)))
for q in range(0, k, bk):
acc += a[i:i+bm, q:q+bk] @ b[q:q+bk, j:j+bn]
out[i:i+bm, j:j+bn] = acc
return out
The 8×8 thread microtile needs 64 scalar accumulators before accounting for addresses and temporaries. Larger reuse tiles may reduce global traffic but consume enough registers/shared memory to reduce residency or spill. Occupancy and Register Pressure explains why a larger tile can lose. Asynchronous copies can overlap the next load with current computation, but their completion and buffer-reuse dependencies still need synchronization. Adding stages increases storage; it is not automatically a speedup.
Put peak utilization on a defensible denominator
Dense square GEMM: M = N = K = 4,096
Work = 2MNK = 137,438,953,472 FLOPs
Hypothetical measured time = 0.2 ms = 0.0002 s
Throughput = work / time = 687.2 × 10^12 FLOP/s
Assumed matching dense peak = 989 × 10^12 FLOP/s
Peak fraction = 687.2 / 989 = 69.5%
Check: the resulting throughput is below the assumed peak.
Changing to a sparse peak without changing the workload would make this comparison invalid. These inputs illustrate the calculation, not a device benchmark.
With BF16 A, B and C, reading both inputs and writing output once requires 100,663,296 bytes, or 96 MiB. The corresponding algorithm-level intensity is 1365.3 FLOP/B, excluding extra reads, intermediate state and output accumulation from a nonzero beta term. This minimum traffic is not the same as a particular kernel's achieved intensity. Use Nsight Compute to profile its actual memory traffic, matrix instruction activity, stalls and tail waves before calling it compute-bound.
Choose tile sizes by representative M, N, K, layouts and epilogue. Keep an optimization only if correctness and full-operation latency improve; reverse a larger tile or extra pipeline stage when spills, underfilled blocks or shared-memory pressure erase its reuse benefit. Small or skinny shapes may favor different tiles or split-K, whose reduction adds work. There is no rule that tiling stops helping whenever one dimension is small.
What interviewers probe next
- What crosses the K-loop boundary? The partial output must survive every slice.
- Where does each reuse ratio apply? Block staging, shared-memory requests and minimum HBM bytes are distinct.
- How do you validate? Compare against a higher-precision reference on ragged, empty and multi-K-tile shapes; inspect errors and event timing after warmup on the target GPU.
Common mistakes
- Resetting accumulators inside the K loop or putting a required barrier behind a non-uniform edge condition.
- Counting BF16 as one byte or comparing dense work against a sparse peak.
- Multiplying reuse factors from different memory levels into a fictional HBM speedup.
Key takeaways
- A 128×128×32 BF16 block slice has a 64 FLOP/B input-staging ratio.
- An 8×8 scalar FP32 microtile has a 2 FLOP/B shared-operand ratio, with at least 64 accumulator values.
- Peak percentage needs an actual timer and matching arithmetic mode; 70% is not a portable pass/fail rule.
See NVIDIA's matrix multiplication discussion for the staged-reuse mechanism.