← Distributed Training
Advanced
Expert Parallelism for MoE
Expert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.
This concept is part of Premium, which includes all concepts, course lessons and answers.
Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Distributed Training & ParallelismExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.→Distributed Training & ParallelismYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?→AI Infrastructure System DesignDesign a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.→Networking, Interconnects & StorageA mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?→AI Infrastructure System DesignDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.→Napkin Math, Cost & CapacityA MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?→