Explain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.

Every MoE layer routes each token to eight experts that live on other GPUs, twice per layer, forward and back. The bytes per rank per layer for DeepSeek-V3, why that all-to-all runs close to one-to-one against compute, and why a single hot expert stalls every rank waiting at the combine.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEA mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.

Foundational

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsA mixture-of-experts model can be split two ways and the choice changes everything. Tensor parallelism shards each expert across GPUs, which keeps every GPU busy and reads every expert's shard on every token. Expert parallelism gives whole experts to whole GPUs, which reads only the selected experts but requires an all-to-all to route tokens to them and back. The all-to-all is the cost, its backend is a configuration choice matched to the interconnect, and expert load imbalance is what actually limits the result.

Foundational

🧮 Open Weights & Serving Engines

Open-Weights Models of 2026The open-weights frontier moved from dense models of tens of billions of parameters to sparse mixtures of experts measured in trillions, and the serving problem changed with it. As of September 2026 the releases an infrastructure engineer is asked about are Z.ai's GLM-5.3 at 753B, Moonshot's Kimi K3 at 2.8T, and DeepSeek's V4 family. What matters for deployment is not the headline count but three other numbers: active parameters per token, the attention design, and the format the weights actually shipped in.

Foundational

🕸️ Distributed Training

Collective Communication PrimitivesAll-reduce, all-gather, reduce-scatter, all-to-all and broadcast are the five operations every parallelism strategy is built from, and each has a fixed per-rank traffic cost you can compute before a job runs. Knowing those volumes for a named model is how you decide whether a layout is compute-bound or waiting on the network.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?Next in this trackYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.Next in this trackExplain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?

FEDITOR'S NOTE

Scored on counting the all-to-alls (four per layer per step), on costing them in bytes against the rank's compute, and on explaining that the collective finishes when the most loaded expert finishes.

DISCUSSION · 0

No comments yet — be the first to share your approach.