A mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEExpert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.

Foundational

🔌 Networking & Storage

NCCL and Collective AlgorithmsNCCL implements GPU collectives using algorithms, protocols and channels over available hardware paths. Learn what each choice changes, how to read bandwidth results, and which evidence separates a communication problem from late application work.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeNVLink, NVSwitch and PCIe provide different GPU communication paths. Learn how their topology and bandwidth affect the placement of parallel work.

Foundational

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsCompare tensor and expert placement without confusing either with sparsity. Derive logical dispatch traffic and redundant-expert memory, then choose backends and balancing from per-rank measurements.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackCheckpoints and datasets both live on storage. Why does one system sized for both usually serve neither well?Next in this trackSeveral unrelated jobs slow down at once and nothing is dropping packets. How do you confirm a pause cascade and find where it started?Next in this track8,192 GPUs exceeds what a two-tier Clos of 64-port switches supports. Compare adding a tier against using higher-radix switches.

FEDITOR'S NOTE

State token ownership and routing assumptions, separate off-rank from cross-node bytes and compare actual dispatch/combine behavior rather than advertised bandwidth.

DISCUSSION · 0

No comments yet — be the first to share your approach.