Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Distributed Training & Parallelism / 13
hardDeepSeekMistralMeta
Explain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.
Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.
Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
Work through the numbers
Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Advanced
Expert Parallelism for MoEExpert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.🕸️ Distributed Training🔒 Premium
Foundational
Expert Parallel and All-to-All BackendsCompare tensor and expert placement without confusing either with sparsity. Derive logical dispatch traffic and redundant-expert memory, then choose backends and balancing from per-rank measurements.🧮 Open Weights & Serving Engines
Advanced
Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.📐 AI Systems Design🔒 Premium
Foundational
Open-Weights Models of 2026Compare current Chinese open-weight models by the artifact you can deploy: weight bytes, attention state, supported execution paths, and evaluation conditions. A dated guide to GLM, Kimi, DeepSeek, Qwen, and MiniMax, with worked sizing checks that separate parameter counts from GPU capacity.🧮 Open Weights & Serving Engines
UP NEXT ON YOUR JOURNEY
Next in this trackYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?Next in this trackYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.Next in this trackExplain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?
FEDITOR'S NOTE
Trace dispatch and return for one token, then count assignments across the group. Separate logical payload, remote bytes and the rank that limits completion.
DISCUSSION · 0
No comments yet — be the first to share your approach.