expert parallelism

Routing hands each expert a different number of tokens, so the layer is a list of ragged matmuls rather than one. Three separate costs a loop pays, the measured share of compute that skewed routing wastes on tile padding, and what a single grouped launch fixes and what it cannot.

Every MoE layer routes each token to eight experts that live on other GPUs, twice per layer, forward and back. The bytes per rank per layer for DeepSeek-V3, why that all-to-all runs close to one-to-one against compute, and why a single hot expert stalls every rank waiting at the combine.

Two all-to-alls per layer across sixty layers is over a hundred collectives on the critical path of every decode step, each one small and latency-bound. The per-GPU volume worked out, the time on each link type, and why a single large NVLink domain changes the design rather than just improving it.

A trillion parameters with 32 billion active per token: the weights need a rack, the compute needs a fraction of it, and the all-to-all between them decides everything. The memory plan across 72 GPUs, the decode step from bandwidth and two all-to-alls, the compressed attention cache, and what imbalance costs.

The first question is whether 1T is total or active, because storage follows one and compute follows the other. The state budget, the layout that falls out of it, why expert parallelism belongs inside NVLink, and the two failure modes a dense-model plan does not have: router imbalance and all-to-all congestion.