TL;DR: Compare SXM and SXM separately on bandwidth, memory capacity and measured application performance. Their advertised dense BF16 peaks are similar; H200's nominal HBM bandwidth is about 1.43 times higher and capacity about 1.76 times larger. Those ratios can improve a memory-limited workload or change its placement. They do not establish a universal decode speedup, concurrent-sequence count or hourly-price premium.
How to approach it
First pin the SKU (the exact product variant, since SXM and NVL versions differ), precision, model and latency target. Split the current step into the portion a faster memory system could improve and the portion it cannot. Then test whether the larger memory changes batch size or sharding. Close with cost per useful completed workload and the price or workload change that reverses the choice.
A strong answer
You are offered an H200 replacement for an H100 serving pool. The quoted node is more expensive, but the workload spends part of each step waiting for collectives. Paying for faster HBM will not accelerate that whole step. Start with the measured breakdown rather than applying a marketing ratio to tokens per second.
Compare like-for-like product specifications
NVIDIA's H100 specifications and H200 specifications distinguish SXM and NVL variants. For the SXM comparison here, use 80 versus 141 GB HBM and 3.35 versus 4.8 TB/s. Both list 1,979 BF16 Tensor Core TFLOPS with sparsity, about 989 TFLOPS dense after dividing by two. Keep that convention visible; sparse and dense figures are not interchangeable.
GPU Generations: to Blackwell provides the broader progression. Similar peak compute does not imply every compute-heavy kernel runs identically: clocks, power settings, library versions and memory behavior still need measurement. H200 remains a Hopper-generation part; it is not a Blackwell compute upgrade.
Nominal SXM ratios:
HBM bandwidth = 4.8 / 3.35 ≈ 1.433
HBM capacity = 141 / 80 = 1.7625
Toy H100 serial step = 8 ms HBM-limited work + 2 ms unchanged work = 10 ms
Toy H200 step = 8 / 1.433 + 2 ≈ 7.58 ms
speedup = 10 / 7.58 ≈ 1.32
sanity: unchanged work keeps this below the 1.433 bandwidth ratio.
This example assumes the HBM-limited portion scales perfectly with the nominal bandwidth and that the two terms are serial. Real overlaps and dependency chains can differ. Roofline Model identifies a useful bound; a trace establishes which part of the application approaches it. Prefill, decode and training each contain several kinds of operations, so none should be assigned one bottleneck without checking.
Derive a decode bound without calling it a benchmark
For an illustrative workload, use 70.6 billion parameters stored in BF16. Its key/value (KV) cache has 80 layers, eight KV heads and head dimension 128, with both keys and values stored in BF16. At 4,096 cached positions:
weights = 70.6e9 × 2 = 141.2 GB decimal
KV per sequence = 2 × 80 × 8 × 128 × 2 × 4,096
= 1,342,177,280 bytes ≈ 1.342 GB
batch32 logical read = 141.2 + 32 × 1.342 ≈ 184.15 GB
ideal eight-GPU HBM time:
H100 = 184.15 GB / 26,800 GB/s ≈ 6.87 ms
H200 = 184.15 GB / 38,400 GB/s ≈ 4.80 ms
sanity: these are payload-time floors, excluding communication and other work.
The sum assumes balanced sharding, one logical weight read and the stated KV traffic per step. It does not measure effective bandwidth, cache behavior, kernels or scheduling. More memory can support a different batch, but that batch reads more KV and may exceed the latency target. Do not multiply a bandwidth ratio by a capacity ratio and label the result throughput.
Capacity changes are often discontinuous
Subtract weights and measured runtime allocations from usable memory on every rank. With an illustrative 6 GB non-KV reserve per GPU and perfectly balanced TP8 weights, the nominal KV remainder is 80 - 17.65 - 6 = 56.35 GB on H100 and 141 - 17.65 - 6 = 117.35 GB on H200. That is about 2.08 times the remainder, not 1.76. The reserve is a hypothetical input, not a universal engine footprint. Paging, replicated tensors and uneven ranks can lower the admitted count.
A 405B BF16 weight payload is roughly 810 GB. Eight 80 GB devices cannot hold that payload; eight 141 GB devices have a nominal 1,128 GB total. The latter passes a payload-only test, leaving 318 GB before KV, workspaces and other allocations. It still needs per-rank validation. Reducing a required second node can materially change both communication and cost.
Training needs its own state recipe. If a 70B run requires 16 bytes per parameter of static state, that is 1,120 GB, not 1.13 TB for this exact parameter count. Eight H200s leave only 8 GB nominal beyond that payload, and an 80% memory budget provides just 902.4 GB. It does not fit the stated headroom, let alone uncounted activations. Different optimizer state, sharding or recomputation may change the result; larger HBM is not a blanket promise that one node trains the model.
Buy measured useful work
If a hypothetical quote is 1.28 times the H100 node-hour price, the toy 1.32 speedup gives only about 1.03 times useful work per dollar, before migration and operational costs. That price ratio is an example, not a current market rate. Benchmark the same quality, context distribution and p99 latency constraint, including failures and reserved idle capacity.
Choose H200 when measured goodput per total cost improves, or when the larger usable memory enables a valuable placement. Keep H100 when that benefit does not pay its actual premium. Revisit the decision if longer context, a new model or reduced activation recomputation changes the memory constraint. Bandwidth-Bound Decode Throughput explains the traffic calculation; the final purchase needs an application result.
What interviewers probe next
- "Can H200 speed up training?" Yes, if memory traffic, capacity or a changed micro-batch/recomputation policy is limiting. Similar compute peaks alone do not settle it.
- "Why can decode gain less than 1.43 times?" Collectives, launches and other unchanged work dilute the HBM improvement; effective bandwidth may scale differently too.
- "Why can capacity help more than its ratio?" Fixed weight allocations leave a larger ratio of free space, and fitting one fewer node creates a placement change.
Common mistakes
- Treating a bandwidth-only floor as measured tokens per second.
- Claiming a 16-byte 70B training state fits eight H200s with 20% headroom.
- Multiplying capacity and bandwidth ratios without accounting for the larger batch's traffic and latency.
Key takeaways
- SXM nominal ratios are 1.43 for bandwidth and 1.76 for total memory; application speedup is workload-specific.
- An eight-plus-two millisecond serial example improves by about 1.32 times when only HBM work scales.
- 1,120 GB of static state exceeds an eight-H200 80% memory budget of 902.4 GB.