TL;DR: Eight H100s do not determine one user count. For each replica (one complete running copy of the model, here spread across GPUs), subtract measured non-cache allocations on every rank, size KV for retained tokens plus allowed growth, and use the limiting rank. Then benchmark latency at feasible concurrency. In the worked TP8 example, a 38 GiB limiting cache budget holds 243 current 4,096-position sequences, but only 194 with 1,024 additional positions reserved; neither count certifies the latency target.
How to approach it
Clarify whether “users” means signed-in people, queued requests or actively generating sequences. Ask for prompt and output lengths, cache format, replica layout and token-latency targets. Derive memory feasibility per GPU, including growth, before testing throughput. Compare TP layouts using actual allocations and measured service behavior.
A strong answer
A replica admits many short prompts successfully. As they generate together, their caches grow and admission starts failing. The weights still fit. The plan counted the requests at arrival, but did not budget the state they would accumulate.
Start with KV Cache Sizing. NVIDIA's Llama 3.1 70B configuration has 80 layers, 8 KV heads and a head dimension of 128. Assume a BF16 cache regardless of the linked checkpoint's weight format. Its raw cache payload is:
2 × 80 layers × 8 KV heads × 128 elements × 2 bytes
= 327,680 bytes = 320 KiB per retained position
A rank is a worker attached to a GPU. For this example, tensor parallelism shards the eight KV heads across eight ranks without replication, so each position needs 40 KiB per rank. That layout must be verified in the engine. TP8 does not create a general pool in which any full GPU can borrow unused memory from another.
For weights, about 70.6 billion parameters at two bytes each gives a rough 141.2 GB BF16 payload. Actual loaded tensors, replication and alignment determine each rank's allocation. Subtract those allocations, activation peaks, graph pools, communication buffers and deliberate headroom from available memory. Do not assume a fixed 10% reserve covers every configuration. vLLM's tuning guide describes how memory and scheduling settings interact.
Suppose the resulting illustrative cache budgets are 40 GiB on seven ranks and 38 GiB on one. These are assumed budgets after non-cache reservations, not measurements from this repository. The 38 GiB rank limits the replica.
Current length: 4,096 retained positions
per-sequence payload per rank = 4,096 × 40 KiB = 160 MiB
current-length ceiling = floor(38 × 1,024 / 160) = 243
Reserve another 1,024 positions per request:
planned length = 5,120 positions
per-sequence reservation = 5,120 × 40 KiB = 200 MiB
growth-reserved ceiling = floor(38 × 1,024 / 200) = 194
Sanity check: 194 reservations consume 38,800 MiB, below the 38,912 MiB budget. A 195th would require 39,000 MiB and fail on the limiting rank. The extra capacity on the other seven ranks does not rescue it.
The next request needs its shard on every participating GPU. This drawing makes the limiting rank visible before any fleet-wide memory total can hide it.
Requests with different lengths need different reservations. Add those reservations rather than treating the average length as a guarantee. If blocks hold 16 positions, round each request's planned length up to a multiple of 16 before adding it; the example lengths already align. With prefix sharing, count the physical blocks that actually exist and the private copies that future writes may require. Adding every request's logical tokens would count shared state more than once.
| Admission check | Evidence needed |
|---|---|
| Does the model load? | Actual per-rank non-cache allocation |
| Can requests finish within their cap? | Cache reservation, block rounding and growth policy |
| Do users get timely tokens? | Measured TTFT and token-gap distributions |
| Does a different layout help? | Allocation and service measurements for that layout |
Memory feasibility does not certify speed. For an optimistic traffic check, assume one 141.2 GB weight pass and one full BF16 cache read per sequence, balanced across eight H100 SXM GPUs at 3.35 TB/s each. At batch 128 and 4,096 positions, the modeled HBM-time floor is about 11.68 ms. At 5,120 positions it becomes 13.28 ms, before collectives and other costs. Thus a 12 ms token-gap target is not established by the initial memory fit or even by the earlier, shorter-context floor. Bandwidth-Bound Decode Throughput explains the assumptions behind this estimate.
I would benchmark several feasible concurrency settings over the expected prompt/output mix, including long-output bursts. Record p99 inter-token gaps, TTFT, completed output and preemptions. Increase admission only while those targets and growth headroom hold. Reserving the maximum output cap is conservative; an engine can instead admit dynamically with a defined preemption or recomputation policy, but its latency consequences must be tested.
Two TP4 replicas duplicate the weight payload, costing roughly another 141.2 GB across the node before changes in runtime allocation. Each replica has its own cache budgets. They can isolate scheduling and rolling updates, but sharing one host still shares a failure domain. Neither “half the communication cost” nor “higher throughput per GPU” follows from halving the rank count; topology, message size and implementation determine the result.
Finally, active generation is not the same as registered users. For an illustrative steady arrival rate of 20 requests/s and mean active-generation duration of 6 seconds, Little's law gives 120 generating requests on average. It does not bound a burst or include a waiting queue unless that queue is part of the measured duration. Size the service against the relevant population and its tails.
What interviewers probe next
- “What if prompts are 32,768 positions?” Adding the same 1,024-position allowance gives 33,792 positions, or 1,320 MiB per sequence per rank. The 38 GiB budget then holds 29 reservations, before latency checks.
- “Does FP8 double concurrency?” It halves raw cache value bytes if the cache itself uses FP8, but scales, allocations, other states and latency remain. Weight-only FP8 does not imply FP8 cache; evaluate quality and backend support separately.
- “Why not divide all free memory by the total cache?” A sharded request needs capacity on every participating rank. Uneven budgets or replication invalidate an aggregate-only division.
- “Should I choose pipeline parallelism instead?” Recompute per-stage memory and measure pipeline scheduling, bubbles and communication. Neither TP nor PP is universally best solely because the weights fit on one node.
Common mistakes
- Counting current prompt state as enough memory for a whole generation.
- Treating tensor-parallel caches as an unrestricted shared pool.
- Turning a peak-bandwidth floor into a promised latency or user count.
- Calling two replicas on one host independent hardware failure domains.
Key takeaways
- The example admits at most 194 growth reservations on its limiting rank, versus 243 current-length payloads.
- Count every rank, every retained position and the declared growth policy.
- A fitting batch still needs a workload-specific latency and preemption check.
- User population, queued requests and active generation are different capacity measures.