b200

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

Decode throughput is bytes per step over bandwidth, price is dollars per hour, so tokens per dollar for a 70B falls out of a spec sheet in ten lines. In bf16 the three generations are within 10% per dollar; fp8 pulls the H100 ahead, and fp4 on the B200 doubles it again. The chain, the table and the caveats.

Decode is bandwidth-bound, so the FP8 FLOPS number that dominates the marketing does not move it. Where the B300 pays is capacity, and capacity converts into throughput through batch size rather than directly. The arithmetic that decides it, and the case where the B200 wins.

8 TB/s over the bytes per step: 57 tokens per second single-stream in bf16, 113 in fp8, 227 in fp4, and the batch curve on a single card that now holds the whole model. Where the curve bends and what caps it.