TL;DR: Memory sets the ceiling: 8 × 80 GB with 10% headroom leaves 435 GB after 141 GB of bf16 weights, which is about 324 sequences at 4k context or 40 at 32k; fp8 weights and fp8 KV push that to roughly 750 at 4k. The latency SLO sets the operating point below the ceiling, because a decode step at batch 256 takes about 18 ms. TP8 shares the KV pool across all cards; two TP4 replicas hold two copies of the weights and lose 141 GB of KV budget to gain independent failure domains and half the all-reduce cost.
How to approach it
Ask three things: context length distribution (the number changes 8x between 4k and 32k), the TPOT SLO (it caps batch below the memory ceiling), and whether fp8 is acceptable. Then say the order: weights, then headroom for activations and CUDA graphs, then the KV budget, then the KV per sequence at the stated context, then the division, then a check that the resulting batch meets the SLO. Say the TP question last, because it changes the budget only through weight duplication.
A strong answer
A typical situation: a capacity plan is written from a GPU count and a hope. The answer is four lines of arithmetic, and it changes by a factor of two depending on one flag nobody set deliberately.
Start with Model Memory Footprint and KV Cache Sizing, in that order:
inputs (Llama 3.1 70B, 8 × H100 SXM 80 GB, 3.35 TB/s each)
weights bf16 = 70.6e9 × 2 B = 141.2 GB
weights fp8 = 70.6e9 × 1 B = 70.6 GB
KV per token bf16 = 2 × 80 × 8 × 128 × 2 = 327,680 B ≈ 328 KB
KV per token fp8 ≈ 164 KB
usable memory = 8 × 80 × 0.9 = 576 GB (10% for activations, workspace, graphs)
KV budget bf16 = 576 - 141.2 ≈ 435 GB
KV budget fp8 = 576 - 70.6 ≈ 505 GB
sequences = KV budget ÷ (KV per token × context)
bf16, 4k: 435 GB ÷ (328 KB × 4,096 = 1.34 GB) ≈ 324
bf16, 8k: 435 GB ÷ 2.68 GB ≈ 162
bf16, 32k: 435 GB ÷ 10.7 GB ≈ 40
fp8 W + fp8 KV, 4k: 505 GB ÷ 0.67 GB ≈ 753
sanity: fp8 roughly doubles the KV budget's reach because it both frees 70 GB and halves KV per token
That is the ceiling, not the operating point. The batch also sets per-token latency, because every decode step reads the weights plus every running sequence's KV:
step time ≈ (W + B × KV per seq) ÷ aggregate bandwidth (8 × 3.35 = 26.8 TB/s)
bf16, 4k, B = 64: (141.2 + 85.8) ÷ 26.8 ≈ 8.5 ms → 7,500 tokens/s
bf16, 4k, B = 128: (141.2 + 172) ÷ 26.8 ≈ 11.7 ms → 10,900 tokens/s
bf16, 4k, B = 324: (141.2 + 434) ÷ 26.8 ≈ 21.5 ms → 15,100 tokens/s
fp8, 4k, B = 256: (70.6 + 172) ÷ 26.8 ≈ 9.1 ms → 28,200 tokens/s
sanity: filling memory (B = 324) costs 21.5 ms per token; with a 15 ms TPOT p95 SLO the
operating batch is nearer 180, and the spare memory is headroom for long contexts
So the answer to "how many users" is two numbers: the memory ceiling and the SLO-bounded batch. With a 15 ms TPOT SLO in bf16, about 180 concurrent 4k conversations; with fp8 and the same SLO, roughly 400, at which point the batch approaches the compute ridge (295 on H100) and the step stops being bandwidth-bound.
Where TP degree enters. With TP8 there is one copy of the weights and one shared KV pool. With two TP4 replicas there are two weight copies (282 GB) and the KV budget drops to 576 - 282 ≈ 294 GB, so 4k concurrency falls from 324 to about 220. What two replicas buy: an all-reduce over four ranks instead of eight per layer (two per layer, 160 per step), which at NVLink speeds is about 30% less communication time per step; independent failure and rollout domains; and a KV pool that is not shared, so one replica's long-context tenant cannot starve the other. TP8 wins on capacity and on batch-1 latency, TP4 × 2 wins on isolation and the throughput per GPU at moderate batch. Pipeline parallelism is the wrong tool here; it adds bubbles and buys nothing on a node that fits the model.
The reversal condition: fp8 weights and an fp8 KV cache roughly double every number below, so a deployment that does not fit in bf16 often fits comfortably one precision down, and the question becomes an evaluation project rather than a hardware one.
Concurrency in users is more than sequences in flight. If each user sends a request every 30 s and a request lives 6 s (1,000-token prompt at 45 ms plus 300 tokens at 12 ms plus queue), one running slot serves about 5 users, so 180 slots is roughly 900 active chat users.
What interviewers probe next
- "What if half the traffic is 32k?" Budget in tokens, not sequences: 435 GB ÷ 328 KB ≈ 1.33 M cached tokens shared across mixed contexts; a 32k session costs 8 of the 4k slots, and the scheduler needs a per-tenant cap or the long tail blocks admission.
- "Would H200s change this?" Yes, twice over: 141 GB per card gives 1,015 GB usable, so the bf16 KV budget rises to 874 GB (about 650 sequences at 4k), and 4.8 TB/s cuts the step time by 30%.
- "Why leave 10% headroom?" Activations for the prefill chunk, CUDA graph buffers, sampling workspace and NCCL buffers; vLLM's gpu_memory_utilization defaults near 0.9 for this reason, and setting 0.98 produces OOM under a long-prompt burst.
Common mistakes
- Dividing 640 GB by the weight size and reporting "it fits" as the answer.
- Forgetting that the KV cache is read every step, so filling memory also fills the step time.
- Counting attention heads instead of KV heads.
- Proposing TP16 across two nodes to "get more memory" when the problem was never the weights.
Key takeaways
- Budget order: usable memory, minus weights, equals KV budget; divide by KV per token × context.
- 8 H100s in bf16 hold about 324 sequences at 4k and 40 at 32k; fp8 weights and KV roughly double reach.
- Step time = (weights + batch × KV per sequence) ÷ aggregate bandwidth; the SLO, not memory, sets the batch.
- TP8 pools KV; two TP4 replicas cost 141 GB of budget for isolation and cheaper all-reduce.