TL;DR: Memory-bound, until the batch is large. Each decode step does 2 FLOPs per parameter per sequence and reads each parameter once, so with bf16 weights the arithmetic intensity is 2 × batch ÷ 2 bytes = batch FLOP per byte. The H100 ridge is 295, so a batch of 1 sits at 1 FLOP/B (0.3% of peak) and a step for Llama 3.1 70B takes 141 GB ÷ 3.35 TB/s = 42 ms per GPU-equivalent. Batching moves the point right almost for free until the KV cache reads, which grow with batch and context, dominate the bytes; at 4k context in bf16 the intensity cannot exceed about 108 no matter the batch.
How to approach it
Ask for the model, the precision and the context length, since each changes a term. Write the FLOPs per step and the bytes per step for one sequence, take the ratio, and compare it with the ridge point. Then add sequences and show how the ratio moves, and add the KV cache term to show where the movement stops. Give a step time in milliseconds at each stage so the interviewer hears tokens per second, not just a classification. Close with the decision: what batch to run and why fp8 changes the answer less than people expect.
A strong answer
A typical situation: a serving fleet reports 100% GPU utilization and 3 TFLOPS of useful work, and a capacity plan is written on the first number. Decode at batch 1 reads every weight to produce one token, and the ratio below says so before anyone opens a profiler.
Memory-Bound vs Compute-Bound Kernels is decided by one ratio, and for decode every term in it is countable.
Per generated token, a dense model does 2 FLOPs per parameter (one multiply, one add, for each weight). Per step, the weights must be read once from HBM regardless of how many sequences are in the batch, because a step for a batch is one matrix multiply per layer with the batch as the M dimension.
model: Llama 3.1 70B, N = 70.6e9 parameters, bf16 weights = 141 GB
batch B = 1:
FLOPs per step = 2 × N = 141 GFLOP
bytes per step = N × 2 B = 141 GB
I = 141e9 ÷ 141e9 = 1 FLOP/B
step time (bandwidth-bound) = 141e9 ÷ 3.35e12 = 42 ms (on one H100's bandwidth)
achieved = 141e9 ÷ 0.042 = 3.4 TFLOPS = 0.34% of 989
sanity: 42 ms per token is 24 tokens/s, the familiar single-stream speed of a 70B split over
a node when you ignore communication; a full 8-GPU node has 8 × 3.35 = 26.8 TB/s,
so the step drops to 5.3 ms and about 190 tokens/s
batch B:
FLOPs = 2 × N × B; bytes ≈ N × 2 (weights, the same)
I = 2NB ÷ 2N = B FLOP/B
ridge on H100 (bf16) = 989 ÷ 3.35 = 295, so decode reaches compute-bound at B ≈ 295
with fp8 weights: I = 2NB ÷ N = 2B, ridge for fp8 compute = 591, so B ≈ 295 again
That last line is worth saying in the room: fp8 halves the bytes and doubles the compute peak, so the batch at which decode crosses the ridge barely moves; what fp8 buys is a 2x shorter step at every batch, not a change in regime.
The weight-only chain overstates how far batching can go, because every sequence brings its own KV cache and the attention kernel reads all of it each step. The KV bytes scale with batch, so they do not amortize:
KV per token (Llama 3.1 70B, bf16) = 2 × 80 layers × 8 KV heads × 128 head dim × 2 B = 320 KB
KV per sequence at 4,096 context = 320 KB × 4,096 = 1.31 GB
bytes per step = weights + B × KV per sequence = 141 GB + B × 1.31 GB
FLOPs per step ≈ 2NB = 141 GFLOP × B (attention FLOPs add ~ 8%, ignored here)
I(B) = 141B ÷ (141 + 1.31B)
B = 32: 4,512 ÷ 183 = 25 FLOP/B step = 183 GB ÷ 3.35 TB/s = 55 ms
B = 128: 18,048 ÷ 309 = 58 FLOP/B step = 92 ms
B = 512: 72,192 ÷ 812 = 89 FLOP/B step = 242 ms
B → ∞: 141 ÷ 1.31 = 108 FLOP/B, the asymptote
sanity: 108 is below the 295 ridge, so at 4k context in bf16 no batch makes a 70B decode
compute-bound on H100; the attention reads take over from the weight reads at
B = 141 ÷ 1.31 ≈ 108 sequences, where the two terms are equal
Two things follow. First, the useful regime is where the step time is still dominated by the fixed weight read, roughly B < 100 at this context, because there each added sequence is nearly free: batch 32 costs 55 ms per step and produces 32 tokens, batch 1 costs 42 ms and produces 1. Second, the levers that matter at long context are on the KV term: fp8 KV halves 1.31 GB to 0.66 GB and lifts the asymptote to 216; GQA already did this once (8 KV heads instead of 64); MLA in DeepSeek-V3 does it again. The attention kernel itself is a separate memory-bound problem at intensity around 8 FLOP/B with GQA, because each sequence's cache is read by that sequence alone.
The serving decision that comes out of this: run decode at the largest batch the TPOT target allows, because throughput rises almost linearly with batch until the KV term takes over. On an 8-GPU H100 node with bf16 weights (17.6 GB per GPU) and a 12 ms TPOT budget, the step is 17.6 GB ÷ 3.35 TB/s = 5.3 ms of weights plus B × 1.31 GB ÷ 26.8 TB/s = B × 49 µs of KV, so B ≈ 137 fits inside 12 ms; that node produces about 11,000 tokens/s. The reversal is short-context, small-model serving on a large-batch offline job, where the batch can climb past the ridge and prefill-style compute limits appear; there, fp8 compute and tensor-core efficiency become the levers.
Prefill, by contrast, is compute-bound from the first request: a 2,000-token prompt is a GEMM with M = 2,000, intensity in the thousands, and it runs near peak. The reversal condition: prefill, or any batch large enough to push intensity past the ridge at 295. Bandwidth-Bound Decode Throughput carries the same arithmetic forward into tokens per second. The two phases want opposite things from the hardware, which is the entire motivation for disaggregated serving.
What interviewers probe next
- "So why does anyone quote 0.3% utilization as a problem?" It is only a problem if the batch could be higher. At batch 1 the ceiling is 3.4 TFLOPS on one card, and being at 3 TFLOPS is a near-perfect kernel; the fix is scheduling, not CUDA.
- "How does speculative decoding fit?" It converts one memory-bound step into a verification of k draft tokens, which is a GEMM with M = k per sequence: the same weight bytes now do k× the FLOPs, so intensity rises by k. It only pays while the batch is well left of the ridge.
- "What about a MoE model?" Replace N with the active parameters for FLOPs and bytes per step, but note every expert touched by any sequence in the batch is read, so at large batch the bytes approach the total parameters while FLOPs stay at active × B: the intensity gain from batching is smaller than for a dense model.
- "What does H200 change?" 4.8 TB/s and 141 GB: the batch-1 step drops from 42 to 29 ms and the KV asymptote is unchanged, but the memory holds 2x the sequences at the same context.
Common mistakes
- Answering "memory-bound" as a fact about LLMs rather than about a batch size and a context length.
- Counting FLOPs per token as N instead of 2N, or forgetting that the batch shares the weight read.
- Ignoring the KV cache and concluding that batch 300 makes decode compute-bound on H100.
- Believing fp8 turns decode compute-bound; it shortens the step, the crossover batch stays near 295.
Key takeaways
- Decode intensity ≈ batch (bf16) or 2 × batch (fp8) from weights alone; ridge ≈ 295, and the crossover batch is ≈ 295 either way.
- Batch-1 step for a 70B in bf16 is 141 GB ÷ 3.35 TB/s = 42 ms per H100-equivalent; 5.3 ms on a full node.
- KV bytes scale with batch: at 4k context bf16 the asymptotic intensity is 141 ÷ 1.31 ≈ 108, below the ridge; fp8 KV and GQA or MLA raise it.
- Serve at the largest batch that fits the TPOT, since throughput is near-linear in batch until the KV term dominates.