How much faster is an H200 than an H100, really? Which workloads see the gain and which do not?

TL;DR: The H200 is an H100 compute die with HBM3e: 4.8 TB/s instead of 3.35 TB/s (1.43x) and 141 GB instead of 80 GB (1.76x), at the same 989 dense bf16 TFLOPS. Anything bandwidth-bound, which is decode and long-context attention, gets up to 1.43x per step and up to 1.76x the concurrent sequences. Anything compute-bound, which is prefill and most training, gets close to nothing. At a typical 1.28x hourly premium the H200 wins on decode-heavy serving and on models that only fit with the extra memory, and loses on prefill-heavy or training work.

How to approach it

Ask which workload the interviewer means, because "faster" has two different answers here. Write the three datasheet numbers side by side and point out that one of them did not change. Classify the workload as memory-bound or compute-bound using the roofline, and apply the bandwidth ratio only to the memory-bound part. Then bring in the capacity ratio for the fits-or-does-not-fit and batch-size effects, and close with a price-adjusted decision and the condition that flips it.

A strong answer

A typical situation: a fleet moves to H200 expecting the hourly premium to buy a proportional speed-up on everything, and the training jobs come back within noise of where they were. The two parts share a compute die, so the premium buys exactly two ratios and nothing else.

Start from what the two parts share and what they do not. The concept page on GPU Generations: A100 to Blackwell has the full table; the three lines that matter here are:

                H100 SXM     H200 SXM     ratio
bf16 dense      989 TFLOPS   989 TFLOPS   1.00
HBM bandwidth   3.35 TB/s    4.8 TB/s     1.43
HBM capacity    80 GB        141 GB       1.76
on-demand       $2.50/h      $3.20/h      1.28   (hardware.ts defaults, vary by provider)

Same die means the same SMs, the same tensor cores, the same NVLink at 900 GB/s. A kernel right of the ridge point runs at the same speed on both cards. So the first thing to say is that the H200 speedup is a property of the workload, and the roofline tells you which side of the line it is on.

A scenario that makes the split concrete: a team serves Llama 3.1 70B in bf16 with tensor parallelism across an 8-GPU node and is quoted H200 nodes at 1.28x the price. Two phases, two derivations.

Decode, batch 32, 4k context, bf16 weights and KV:

weights per step = 70.6e9 params × 2 B = 141 GB (read once per step, sharded over 8 GPUs)
KV per sequence at 4,096 tokens = 2 × 80 layers × 8 KV heads × 128 dim × 2 B × 4,096 = 1.31 GB
bytes per step = 141 + 32 × 1.31 = 183 GB

node bandwidth: H100 = 8 × 3.35 = 26.8 TB/s     H200 = 8 × 4.8 = 38.4 TB/s
step time:      H100 = 183e9 ÷ 26.8e12 = 6.8 ms  H200 = 183e9 ÷ 38.4e12 = 4.8 ms
speedup = 6.8 ÷ 4.8 = 1.43x, exactly the bandwidth ratio
tokens/s per node: H100 = 32 ÷ 0.0068 = 4,700   H200 = 32 ÷ 0.0048 = 6,700
per dollar: 1.43 ÷ 1.28 = 1.12x more decode tokens per dollar on H200

sanity: intensity at batch 32 is about 25 FLOP/B, far left of the 295 ridge, so
        the bandwidth ratio applies in full and no compute term interferes

Prefill of a 2,048-token prompt for the same model:

FLOPs = 2 × 70.6e9 × 2,048 = 289 TFLOP (attention adds a few percent, ignored)
node peak bf16 = 8 × 989 = 7.9 PFLOPS; at 60% MFU = 4.75 PFLOPS
time = 289e12 ÷ 4.75e15 = 61 ms on either card
speedup = 1.00x; per dollar = 1 ÷ 1.28 = 0.78x, a loss

sanity: a GEMM with M = 2,048 has intensity above 1,000 FLOP/B, right of the ridge,
        so bandwidth does not enter the time at all

The blended answer depends on the traffic mix. If prefill is 30% of GPU time and decode 70%, the H200 node finishes the same work in 0.3 × 1.0 + 0.7 ÷ 1.43 = 0.79 of the time, a 1.27x speedup, which at a 1.28x price is break-even. A chat workload with short prompts and long generations tilts toward H200; a RAG workload that stuffs 20k tokens of context and emits 200 tokens tilts toward H100. Say that the mix is the input the interviewer has not given you, and ask for it.

The capacity ratio is the second effect and it is often the larger one. Memory left after weights is where the KV cache lives:

per GPU after weights (TP8): H100 = 80 - 17.6 = 62 GB     H200 = 141 - 17.6 = 123 GB
minus about 6 GB of activations, CUDA context and fragmentation: 56 GB vs 117 GB
KV per sequence per GPU at 4k (sharded 8 ways) = 1.31 ÷ 8 = 164 MB
max concurrent sequences: H100 = 56 ÷ 0.164 = 341     H200 = 117 ÷ 0.164 = 713

If the service is admission-limited by KV memory, H200 holds 2.1x the sequences, and decode throughput rises almost linearly with batch in this regime. That is a bigger gain than 1.43x and it costs no bandwidth. It also makes long context possible at all: a 128k-token sequence needs 41 GB of KV, which is 5 GB per GPU at TP8; H100 fits about 11 of them and H200 about 23.

The third effect is the model that fits only on one side. Llama 3.1 405B in bf16 is 810 GB. Eight H100s hold 640 GB, so it needs fp8 or a second node; eight H200s hold 1,128 GB, so it fits in bf16 with 318 GB spare for KV. The H200 turns a two-node TP16 or PP2 deployment, which pays cross-node communication on every step, into one node.

The decision: H200 for decode-dominated serving, for anything KV-admission-limited, and for models that fit on one node only with 141 GB per card. H100 for prefill-heavy serving, training and fine-tuning, where the die is the same and the memory is rarely the binding constraint at the batch sizes people train with. The reversal condition on the training side is memory: if the extra 61 GB per card lets you drop activation checkpointing or raise the micro-batch until the GEMMs fill the tensor cores, the H200 can win training too, and you find out by measuring MFU at both settings, or by watching DCGM_FI_PROF_PIPE_TENSOR_ACTIVE rise as the micro-batch grows. Bandwidth-Bound Decode Throughput is the other half of this comparison, and it is where the 1.43x actually lands.

What interviewers probe next

• "Why does not the H200 get faster prefill with all that bandwidth?" Because prefill GEMMs are compute-bound; bytes per FLOP for a 2k-token GEMM are a thousand times lower than the roofline needs to notice bandwidth.
• "Where does the 1.43x fail to materialize even on decode?" When the step includes communication that does not scale with bandwidth: TP8 all-reduce latency is the same on both cards, so at very small batch with 160 all-reduces per step the fixed cost dilutes the gain.
• "You said 1.12x per dollar; is that worth a migration?" Alone, marginal. Combined with the batch gain from 1.76x capacity it is often 1.5x to 2x tokens per dollar, and that is worth it.
• "What does the H200 do for training a 70B?" Little on step time; a lot if the memory lets you shard less. Static Adam state is 16 B per parameter, 1.13 TB for 70B, so FSDP over 8 H100 (640 GB) does not hold it and 8 H200 (1,128 GB) does at 80% headroom. The saved node changes the cost more than the bandwidth does.

Common mistakes

• Quoting a single "H200 is 1.4x faster" and applying it to prefill and training.
• Forgetting the price ratio and calling a 1.43x speedup a win when the per-dollar gain is 1.12x.
• Ignoring the capacity effect, which for KV-limited services is the larger of the two.
• Treating the memory as free KV space without subtracting weights, activations and the CUDA context.

Key takeaways

• H200 = H100 die + HBM3e: bandwidth 4.8 vs 3.35 TB/s (1.43x), capacity 141 vs 80 GB (1.76x), compute identical.
• Decode step time scales with bytes ÷ bandwidth, so up to 1.43x; prefill and training GEMMs scale with FLOPs ÷ peak, so 1.0x.
• At a 1.28x price, decode gains 1.12x per dollar on bandwidth alone; the capacity gain, 2.1x the KV-limited batch, is usually worth more.
• The model that fits on one node only at 141 GB per card (405B bf16, 70B with Adam state) is the strongest H200 argument.