expert parallel

Three things change at once: dense compute more than doubles with an fp4 tensor core, HBM3e reaches 8 TB/s, and NVLink stops at 72 GPUs instead of 8. Work through what each does to tensor-parallel degree, per-step weight reads and expert placement for a 671B-parameter mixture of experts.

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

Expert parallelism reads fewer bytes and pays two all-to-alls, so it loses when the interconnect is wrong or when the routing is uneven. Three candidate causes, the measurement that separates them, and why the third one is the answer more often than the first two.