TL;DR: First choose a layout that holds the model. A BF16 70B weight payload is about 141.2 GB, so it cannot fit on one 80 GB . With ideal TP8 sharding, each GPU reads about 17.65 GB of weights per step: a 5.27 ms weight-read floor, or about 190 tokens/s for one stream before cache traffic and execution costs. This is an upper bound under stated assumptions, not a measured user experience.
How to approach it
First establish weight and cache formats, GPU count and context length. Then estimate one ordinary decode step for one active sequence. Check that each GPU has room for its tensors before dividing traffic by bandwidth. Close by separating the model's ideal floor from measured token gaps and time to first token.
A strong answer
A demo produces a slow stream. Someone divides H100 bandwidth by the model's weight size and announces 24 tokens/s. The division is easy; the deployment it describes is missing. Those weights cannot reside on that one card.
Use Model Memory Footprint to check placement. About 70.6 billion parameters at two bytes each gives 141.2 GB of BF16 weights, before runtime buffers or The KV Cache. For this worked example, assume one tensor-parallel replica across eight H100 SXM GPUs (each GPU holds one eighth of every weight matrix, so each reads only its own eighth per step) with evenly sharded weights and KV heads. Each rank is a worker on one GPU. Verify actual allocations and any replicated tensors in the chosen engine.
NVIDIA lists H100 SXM bandwidth at 3.35 TB/s. Assume each weight is read from HBM once per ordinary decode step, with balanced work and no offload:
weight payload = 70.6e9 × 2 = 141.2e9 bytes
per-rank weight read = 141.2e9 / 8 = 17.65e9 bytes
weight-read floor = 17.65e9 / 3.35e12
= 0.0052687 s = 5.27 ms
weight-only rate ceiling = 1 / 0.0052687 ≈ 190 tokens/s
sanity: 8 × 17.65 GB reconstructs the 141.2 GB payload.
The corresponding dense projection-work floor is roughly 141.2e9 FLOPs divided by eight times 989e12 FLOP/s, about 0.0178 ms at advertised peak. That large gap motivates investigating weight traffic first. It does not prove tensor cores are idle for a particular percentage of wall time. Such a claim needs a trace; ideal compute and memory durations can overlap and neither describes actual utilization.
Add context before presenting a latency estimate. Under the Llama cache dimensions in KV Cache Sizing, BF16 KV needs 327,680 bytes per retained position across the model. At 8,192 positions, a full cache read adds 2.5 GiB per step across the replica. Dividing the combined traffic by 26.8 TB/s gives about 5.369 ms, or a bandwidth-only ceiling of 186 tokens/s. At 32,768 positions it becomes 5.669 ms, about 176 tokens/s.
Even the longer example can have a small payload compared with the eight-card capacity, but available cache space still depends on actual per-rank reservations. Long-context attention also adds arithmetic. A full-step estimate must include that work, activation traffic, collectives and launch or scheduling costs. Avoid adding independent peak-resource times as though every cost were serial; trace which portions overlap and which sit on the critical path.
| Candidate change | What must be checked |
|---|---|
| Lower weight precision | Actual bytes, conversion cost and quality |
| More TP ranks | Per-rank traffic and added communication |
| Speculative decoding | Accepted output per total draft/verify time |
| More active sequences | Aggregate rate and each stream's token gaps |
I would benchmark the existing feasible layout before buying hardware. Record prompt length, output length, warm-up state, engine version and precision, then measure inter-token gaps with the serving benchmark described in vLLM's CLI reference. If a trace shows host gaps or collectives dominate, increasing HBM bandwidth will address only part of the delay. If weight traffic dominates and an evaluated lower-precision format is supported, reducing that traffic is a useful experiment.
Batching may increase aggregate output by sharing weight reads. It does not promise unchanged per-user speed: cache traffic and work grow. Bandwidth-Bound Decode Throughput shows this distinction. Likewise, quantization does not guarantee a 2× speedup, and speculative decoding has no fixed accepted-token count.
What interviewers probe next
- “What does 500 output tokens imply?” With 500 emitted tokens there are 499 measured gaps after the first. Under an illustrative constant 10 ms gap, the streaming portion is 4.99 seconds; add measured time to first token separately.
- “What about a smaller model?” An illustrative 8B BF16 payload is 16 GB. The weight-only floor on one H100 SXM is 4.78 ms, about 209 tokens/s. Check runtime fit, cache and other costs before treating it as service capacity.
- “Can one H100 hold a quantized 70B?” A one-byte-per-weight payload is about 70.6 GB, leaving limited nominal space on an 80 GB card. Scales, protected tensors, KV and runtime allocations decide whether a specific context fits.
- “Is 24 tokens/s wrong?” It is the weight-only bandwidth ratio for one card, but it is not evidence for a resident BF16 one-card deployment or a measured speed on TP8.
Common mistakes
- Estimating speed before checking placement.
- Labeling an ideal traffic floor as measured latency.
- Turning theoretical compute time into an idle-time percentage.
- Counting all output tokens as gaps after the first token.
Key takeaways
- 141.2 GB of BF16 weights does not fit on one 80 GB H100.
- The stated TP8 weight-only floor is 5.27 ms; adding 8k cache traffic gives 5.369 ms.
- Measured token gaps include costs a bandwidth division omits.
- One-stream pace and aggregate serving throughput use different numerators.