blackwell

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

Four bits give eight magnitudes per sign, and the largest step between them is 50%. Block scaling rescues most of the tensor; a single outlier in a block of sixteen flushes its neighbors to zero. Where the precision goes, which layers fail first, and the measurements that separate acceptable from broken.

8 TB/s over the bytes per step: 57 tokens per second single-stream in bf16, 113 in fp8, 227 in fp4, and the batch curve on a single card that now holds the whole model. Where the curve bends and what caps it.