Use a ten-expert example to separate assignments, rank load and dropped routes, then compare balancing controls against quality.
This answer is included with a free account. Sign in to read it and save your progress.
free account · no card needed
← Distributed Training & Parallelism / 14
hardDeepSeekMistral
Your MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?
Use a ten-expert example to separate assignments, rank load and dropped routes, then compare balancing controls against quality.
Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.
Work through the numbers
Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.
The concepts behind this question
Use these concept explanations to review the ideas behind this question.
Advanced
Expert Parallelism for MoEExpert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.🕸️ Distributed Training🔒 Premium
Advanced
Request Routing and Load Balancing for LLMsAn LLM router chooses a compatible replica, weighing queue time against reusable prompt state. Cached work helps only when the extra wait does not erase the saving.📐 AI Systems Design🔒 Premium
Foundational
Consistent Hashing and ShardingConsistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.💻 Coding for Infra
Foundational
Expert Parallel and All-to-All BackendsCompare tensor and expert placement without confusing either with sparsity. Derive logical dispatch traffic and redundant-expert memory, then choose backends and balancing from per-rank measurements.🧮 Open Weights & Serving Engines
UP NEXT ON YOUR JOURNEY
Next in this trackYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.Next in this trackExplain mixed-precision training. Why does fp16 need loss scaling and bf16 not, why keep fp32 master weights, and what changes with fp8?Next in this trackYour training run delivers only 60% of the throughput you projected. How do you find out whether it is compute, memory, network or I/O?
FEDITOR'S NOTE
Clarify the routing denominator and convert expert counts into rank load. Quantify discarded assignments before comparing balancing controls against model quality.
DISCUSSION · 0
No comments yet — be the first to share your approach.