← Distributed Training
Advanced
LatentMoE and Compressed Expert Activations
LatentMoE projects activations into a smaller space for routed experts, then restores the model width. It can reduce dispatch bytes, but extra projections, routing choices and model quality determine the benefit.
This concept is part of Premium, which includes all concepts, course lessons and answers.
Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Distributed Training & ParallelismHalve the experts and top-K, then double expert width. What actually gets cheaper?→Distributed Training & ParallelismExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.→Networking, Interconnects & StorageA mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?→GPU & Accelerator ArchitectureYour TPU matmul width is a multiple of 256. Why does wider sharding introduce padding?→AI Infrastructure System DesignDesign a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.→CUDA, Triton & Kernel EngineeringA mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?→
