TL;DR: More memory can let a replica serve more sequences, but it does not guarantee more tokens per second. Compare the exact server configurations using your model, context lengths and latency targets. Treat the question's 1.4× power ratio as a planning assumption until measured. 's larger memory is useful when it removes a real capacity constraint; attention performance, communication, software support and total cost can also change the result.
How to approach it
First identify what stops the current replica from accepting more useful work. Calculate its memory requirement, then test throughput at the required latency. Keep per-GPU, per-server and per-rack numbers separate throughout the comparison.
A strong answer
Suppose a team serves a 70B model on eight H100s and is considering or B300 servers. A nearly full KV cache and low compute activity suggest questions to investigate. They do not establish the bottleneck: the engine may reserve a large pool, retain reusable prefixes, wait for communication or spend time in memory-intensive kernels.
Start with the exact hardware configuration
The H100 SXM specifications list 80 GB of memory and 3.35 TB/s of bandwidth. NVIDIA's HGX reference architecture lists 180 GB for B200 and 288 GB for B300, with up to 8 TB/s for either. These are reference specifications, not the free memory and sustained bandwidth your serving process will observe. Product pages may use different capacity conventions; record the exact SKU and runtime-visible bytes.
Also read the precision footnotes. The HGX comparison lists 72 PFLOPS of sparse FP8 for an eight-GPU B200 or B300 system. Dense throughput is half: 36 PFLOPS per system, or 4.5 per GPU. Calling 9 PFLOPS per GPU dense doubles the published dense figure. Equal FP8 peaks still do not imply equal attention performance.
Accelerator Selection: to B300 and explains how memory, software and system constraints narrow the choice. For this workload, calculate capacity before interpreting a throughput claim.
Work out what the extra memory can hold
Use an illustrative budget of 180 or 288 decimal GB per GPU, a 92% allocation limit, 10 GB of workspace per GPU and 70 GB of FP8 weights evenly sharded over eight GPUs. Real weights also need scales and metadata; replace these assumptions with measured allocations before sizing a deployment.
For a grouped-query model with 80 layers, eight KV heads, a head dimension of 128 and two-byte KV elements:
KV bytes per token across the whole model
= 2 (K and V) × 80 × 8 × 128 × 2
= 327,680 bytes = 320 KiB
Illustrative KV pool across eight GPUs
B200: 8 × (180 × 0.92 - 70/8 - 10) = 1,174.80 GB
B300: 8 × (288 × 0.92 - 70/8 - 10) = 1,969.68 GB
At 8,192 cached tokens per sequence
each sequence uses 327,680 × 8,192 = 2.68435456 GB
B200: floor(1,174.80 / 2.68435456) = 437 sequences
B300: floor(1,969.68 / 2.68435456) = 733 sequences
sanity: those are memory ceilings under even KV sharding, before block rounding,
imbalance and extra safety margins. They are not safe admission limits.
The nominal memory ratio is 288/180 = 1.6. The remaining pool grows by about 1.677× because the assumed weight and workspace deductions stay fixed. This model assumes eight KV heads divide evenly across eight tensor-parallel ranks. A layout that replicates KV needs different accounting. Quantized weights do not automatically mean quantized KV.
Capacity is only one term in decode time
A useful first model for one dense decode step on each rank is:
bytes read ≈ local weights + batch × context length × local KV bytes per token
memory-time floor ≈ bytes read / sustained local bandwidth
step time also depends on compute, collectives and scheduling
throughput ≈ batch / measured step time
The terms can overlap, so do not blindly add separate worst-case times. This is a way to identify what to measure. At large batches or long contexts, KV reads and attention work can be substantial. At small batches, weight reads often dominate. Bandwidth-Bound Decode Throughput develops that distinction.
For an eight-way tensor-parallel deployment, use each rank's weight and KV shard with that rank's bandwidth. Include the batch's KV traffic before estimating a step-time floor, then measure the remaining compute and communication costs.
The example uses the same eight-way sharding and 8,192-token sequence length.
Memory budget
- 437 → 733 sequences
These are illustrative capacity ceilings after fixed deductions, before further overhead.
Serving result
- Measure step time at the target latency
A larger batch also adds KV traffic and attention work; it may hit compute or communication limits.
Make the purchase conditional on a load test
Replay the same prompt/output-length distribution on both configurations. Sweep concurrency while recording useful output tokens per second, time to first token, inter-token latency, queued work, preemptions, live KV allocation, achieved bandwidth and power at the server. Keep precision, quality criteria, engine version and scheduling settings comparable; also test the best supported configuration for each system and label that comparison separately.
If extra B300 memory eliminates preemption and improves throughput within the latency target enough to lower total cost per useful token, it has a case. If the B200 already meets the target at lower total cost, keep it. Reverse either decision when workload shape, prices, software support or facility limits change. A full cache or one utilization counter alone cannot make that decision.
For the stated power assumption, 1.4× measured power would require more than 1.4× useful throughput to improve tokens per joule at the same measurement boundary. GPU power limits are not server wall power. Include CPUs, networking, cooling, acquisition or rental cost and the number of servers the rack can support. Existing H100 capacity may remain economical even when newer hardware is faster.
What interviewers probe next
- “Does decode always ignore compute?” No. Batch size, attention design, context length and kernel efficiency can make compute or communication limiting.
- “Does compressed KV remove the value of memory?” It reduces one allocation; model weights, larger batches and longer contexts may still use the capacity.
- “Would you separate prefill and decode?” Test whether independent placement outweighs KV transfer and operational costs. It is a deployment choice, not an automatic consequence of this comparison.
Common mistakes
- Mixing sparse with dense FLOPS or per-GPU with whole-server figures.
- Turning a 437-to-733 memory ceiling into a measured throughput ratio.
- Treating allocated cache occupancy as proof that active requests lack capacity.
- Comparing GPU TDP against server wall power.
Key takeaways
- In the reference configuration, 288/180 = 1.6× nominal memory; both bandwidth figures are up to 8 TB/s.
- The worked memory budget holds at most 437 or 733 full 8,192-token sequences before further overheads.
- Choose on useful throughput at the required latency and total cost, then state what evidence would change the choice.