← 🕸️ Distributed Training
Advanced
Expert Parallelism for MoE
A mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Distributed Training & ParallelismExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.→Distributed Training & ParallelismYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?→AI Infrastructure System DesignDesign a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.→Networking, Interconnects & StorageA mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?→AI Infrastructure System DesignDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.→Napkin Math, Cost & CapacityA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?→