TL;DR: Long prefills let many tokens share loaded weights; small decode batches do little work per weight byte. This often makes prefill compute-limited and decode bandwidth-limited. Use the Roofline Model to form a hypothesis, then measure the workload: cache traffic, kernel overhead and queueing can change the answer.
How to approach it
Start with Prefill vs Decode: prompt processing predicts the first token, and ordinary decode predicts subsequent tokens one step at a time. Ask about the model, precision, context lengths, batch size and latency target. Then compare the arithmetic with the bytes moved. Finish with a hardware decision and the measurement that would make you change it.
A strong answer
Suppose a chat service streams slowly even when only one person is using it. Buying a GPU with more advertised FLOPS may disappoint: the model might spend most of each step waiting for weights to arrive from memory.
Arithmetic intensity means operations performed per byte moved. Processing many tokens together can reuse a loaded weight tile across those tokens. A single decoding sequence has only one new token to contribute. The NVIDIA matrix multiplication guide explains why this changes the likely limit, while noting that a roofline comparison simplifies real kernel behavior.
The reused object is a weight tile, a small piece of a matrix. The picture isolates one linear layer, one of the large weight-matrix multiplies that hold most of a model's parameters: more token rows can use those weights before they must be loaded again. The rows are not a claim about a particular kernel tile size.
Use a deliberately simplified example: eight billion dense linear-layer weights stored in BF16, two bytes each, on one SXM 80 GB. The 16 GB of weights fit; KV cache and runtime allocations need their own capacity check. Count roughly two operations per weight per input token. Ignore attention, activation traffic and kernel overhead only for this first estimate.
NVIDIA's H100 specifications give 3.35 TB/s HBM bandwidth and roughly 989 TFLOPS dense BF16 compute for SXM. Do not use the doubled sparse peak for a dense model. Using decimal GB and TB:
P = 8 × 10^9 weights; N = tokens processed together
weight bytes = 2P = 16 × 10^9 bytes
linear-layer work ≈ 2PN operations
weight-only intensity ≈ 2PN / 2P = N operations/byte
hardware ridge ≈ 989 × 10^12 / (3.35 × 10^12)
≈ 295 operations/byte
The ridge is the intensity at which the GPU's arithmetic and its memory would finish at the same time. Below it, the GPU waits on memory. Above it, the arithmetic units are the limit. This weight-only approximation puts a one-token decode step far below the ridge and a 2,048-token prefill above it. It does not establish that a real decode batch of 295 crosses into compute-limited operation: its attention and activation traffic were omitted.
Compute and memory can overlap, so take the larger of their ideal times rather than adding them:
| Tokens | Weight-read floor | Compute floor |
|---|---|---|
| 1 (decode) | 4.78 ms | 0.016 ms |
| 64 (decode) | 4.78 ms | 1.04 ms |
| 2,048 (prefill) | 4.78 ms | 33.13 ms |
For example, reading the weights takes at least 16e9 / 3.35e12 = 0.00478 seconds under this full-weight-stream assumption. The prefill arithmetic takes 2 × 8e9 × 2048 / 989e12 = 0.03313 seconds at peak. These are optimistic limits for counted work, not measured TTFT or token gaps.
Sanity check: multiplying the token count by 64 multiplies the counted arithmetic by 64, while the assumed weight-read volume stays fixed. Real kernels can read weights more than once, and each sequence also brings its own context. The unchanged memory floor is not a promise of unchanged latency.
The next step is a trace and a workload-matched benchmark:
| Observed critical path | Experiment and reason to reverse it |
|---|---|
| Weight movement dominates decode | Test more bandwidth or fewer weight bytes; reverse if KV, overhead or quality loss dominates. |
| Prefill linear kernels dominate | Test more compute; reverse if attention, communication or queueing dominates. |
| Streams pause during prefills | Test smaller prefill chunks; reverse if scheduling cost or prompt delay outweighs the benefit. |
| Most TTFT is queueing | Test capacity or admission changes, then recheck execution latency. |
Benchmark both phases with the same arrival rate and distribution of prompt and output lengths. Record time to first token (TTFT), the gaps between later tokens and completed throughput. TTFT includes time waiting and delivering the response as well as model execution; later token gaps can include scheduling delays. Those measurements explain the hardware requirement more reliably than a prompt-to-output ratio alone. Extra bandwidth may help memory-limited work during prefill too.
What interviewers probe next
- “Why not keep increasing the decode batch?” Each added sequence contributes cache reads, arithmetic and memory allocation. Keep increasing only while measured token gaps and capacity remain acceptable. The 64-token row gives a lower bound, not free throughput.
- “What if most of the prompt is cached?” The uncached work changes. Recompute the estimate for what actually executes, then account for lookup, scheduling and any required suffix processing.
- “Does weight-only quantization double arithmetic intensity?” It reduces the weight-byte denominator, but leaves other traffic and work. Do not silently double the GPU's compute peak unless the executed kernels use the corresponding arithmetic format.
Common mistakes
- Putting 141 GB of BF16 70B weights on one 80 GB GPU without sharding or offload.
- Calling 4.78 ms a measured token gap, or treating the roofline as a complete model of attention.
- Using sparse peaks for dense arithmetic or mixing decimal GB with binary GiB.
- Choosing hardware before distinguishing queueing, model execution and delivery time.
Key takeaways
- The simplified BF16 linear-layer intensity is N operations per weight byte; actual intensity includes more traffic.
- In this example, weight reads have a 4.78 ms floor; 2,048 tokens have a 33.13 ms linear-compute floor.
- Batching improves weight reuse but adds per-sequence work. Verify both throughput and token gaps.
- Choose bandwidth or compute from the measured critical path, and reverse the choice when that path changes.