TL;DR: Small-batch dense decode often warrants investigating weight bandwidth first: weight-only arithmetic intensity is about the batch size with BF16 weights. That shortcut does not classify the whole step. Add context-dependent KV traffic, check per-GPU capacity and communication, then measure the kernels and token gaps before choosing a batch or claiming a compute crossover.
How to approach it
Ask for the model layout, weight and cache precision, active batch and attended lengths. Derive the weight-only ratio, state what it omits, then work a feasible multi-GPU example. Separate a memory-time floor from a measured token gap. Finish with the evidence that would change your optimization choice.
A strong answer
Thirty-two requests are generating together. Their linear layers can share a weight read, but each request also needs attention state for its own history. Increasing the batch reuses weights more; increasing the histories adds work and traffic that this reuse does not remove.
For a dense linear-layer approximation, let N be the weights used, b the sequences taking one ordinary decode step and q the bytes per weight:
linear work ≈ 2Nb FLOP
weight traffic ≈ Nq bytes, assuming one HBM pass
weight-only intensity ≈ 2b/q FLOP per byte
BF16, q = 2: intensity ≈ b
This is an estimate for the linear work, not all model operations. The Roofline Model compares it with a matching peak-to-bandwidth ratio. For the H100 SXM, dense BF16 tensor cores at about 989 TFLOP/s and 3.35 TB/s HBM give a ridge near 295 FLOP/byte. That does not establish a universal “batch 295” serving transition.
Use an illustrative 70-billion-weight dense model, with 80 attention layers, 8 KV heads and head dimension 128. Store weights and cache in BF16. Put one replica across eight 80 GB SXM GPUs. For this estimate assume balanced weight shards and one KV head per GPU, with no replicated payload. Real tensor-parallel layouts need a per-rank check.
weights W = 70 × 10^9 × 2 = 140 × 10^9 bytes
KV bytes per cached position = 2 × 80 × 8 × 128 × 2
= 327,680 bytes = 320 KiB
at S = 4,096 positions:
Q = 327,680 × 4,096 = 1,342,177,280 bytes = 1.25 GiB/sequence
ideal aggregate bandwidth = 8 × 3.35 = 26.8 TB/s
modeled read traffic D = W + bQ
memory-time floor = D / (26.8 × 10^12)
The two cache factors are K and V; KV Cache Sizing explains the dimensions. Here S is the number of positions read by the step. Assume full attention, no shared-prefix traffic savings and one read of each cached K/V value. Exclude new-cache writes, activation traffic, workspace and collectives. Cache hits or repeated reads can make actual HBM traffic differ.
| Batch and attended length | Modeled memory floor |
|---|---|
| 32 sequences, 4,096 positions | 6.83 ms |
| 128 sequences, 4,096 positions | 11.63 ms |
| 32 sequences, 16,384 positions | 11.63 ms |
Four times the batch or four times the context gives the same KV payload in this model. It does not give the same output throughput or arithmetic work. The 128-sequence case produces 128 new tokens per ordinary step; the long-context case produces 32.
Count what the iteration delivers as well as what it reads. These two shapes have equal retained-position totals under the model, but different output numerators.
Check placement before timing. At batch 128 and 4,096 positions, (W + 128Q)/8 ≈ 38.97 GB of weights and KV payload sits on each GPU under the balanced assumption. That leaves room below 80 GB, but the engine still needs workspace, allocation overhead and growing-cache headroom. One 80 GB H100 cannot hold these 140 GB BF16 weights by itself.
Suppose the service needs a 12 ms gap between tokens. The 11.63 ms memory floor leaves only about 0.37 ms for everything this optimistic estimate omitted. It therefore gives little reason to expect batch 128 to meet the target. Collectives, scheduling, kernel execution and less-than-peak bandwidth still consume time. The estimate also adds eight GPU bandwidth peaks, which assumes balanced concurrent reads. Check the slowest rank and the actual overlap before using that aggregate rate.
The old weight-only crossover also misses attention arithmetic. With 64 query heads, an approximate QK plus AV count for one decode token is 4 × 80 × 64 × 128 × S. At 4,096 positions that is about 10.74 GFLOP, in addition to the 140 GFLOP linear approximation, before softmax and other work. Increasing context changes both bytes and work. A ratio that adds KV bytes while ignoring all attention work cannot prove the entire decode step stays memory-bound for every batch.
I would sweep feasible batch sizes over the expected length distribution, record throughput and token-gap tails, and inspect the costly intervals. vLLM's tuning guide discusses the interaction of scheduling, memory and latency; its settings require workload measurements. If HBM traffic dominates, test reuse or smaller representations. If launches or collectives dominate, investigate those first. Memory-Bound vs Compute-Bound Kernels gives the diagnostic distinction. A larger batch loses its case when latency or memory headroom becomes unacceptable.
What interviewers probe next
- “Do FP8 weights guarantee twice the speed?” No. They reduce the weight term, while cache traffic and other costs remain. Weight-only quantization also does not prove the kernel uses FP8 arithmetic.
- “Does speculative decoding multiply useful work by the draft length?” Verification processes candidate tokens, but accepted proposals plus the correction or bonus advance the output. Count draft cost, verification time and emitted output before claiming a gain.
- “Can I replace N with active parameters for MoE?” Use active work for the compute estimate, but traffic depends on the union of experts touched by the batch and their placement or residency. One sequence's active parameter count is not the batch's weight traffic.
- “What happens if only one request has a very long history?” Sum each sequence's attended positions for the traffic model; do not multiply the batch by one average without checking tails, padding and scheduling behavior.
Common mistakes
- Reporting a memory-time floor as a measured per-token latency.
- Using an impossible single-GPU placement or assuming aggregate bandwidth guarantees linear scaling.
- Treating batch size as the whole step's intensity while omitting attention and activation work.
- Choosing the largest batch that fits memory without checking token gaps and cache growth.
Key takeaways
- BF16 weight-only intensity is about
b; it is a useful starting hypothesis with explicit omissions. - The example's cache uses 320 KiB per position, or 1.25 GiB per sequence at 4,096 positions.
- Batch 128 has an 11.63 ms modeled memory floor, which cannot certify a 12 ms token-gap target.
- Select batch size from feasible, measured service behavior; investigate the actual limiting interval.