hard★ EssentialNewOpenAIAnthropicxAI

Design a training cluster for a one-trillion-parameter MoE. Size it, choose the parallel layout, and map it onto the fabric.

The first question is whether 1T is total or active, because storage follows one and compute follows the other. The state budget, the layout that falls out of it, why expert parallelism belongs inside NVLink, and the two failure modes a dense-model plan does not have: router imbalance and all-to-all congestion.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEA mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.

Advanced

📐 AI Systems Design🔒 Premium

Training Cluster Design at 10k GPUsDesign a cluster for training frontier models is the prompt that tests whether a candidate can hold hardware, network, storage, scheduling and reliability in one head at once. The answer is a bill of materials with a reason for every line: how many GPUs and why, how they are grouped into pods, how the fabric connects the pods and what it costs a collective to cross one, how much storage bandwidth the checkpoints and the data loader need, how power and cooling bound the whole thing, and how the failure statistics set the spare pool and the checkpoint cadence. This page derives each line for a 10,240-GPU cluster.

Foundational

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsA mixture-of-experts model can be split two ways and the choice changes everything. Tensor parallelism shards each expert across GPUs, which keeps every GPU busy and reads every expert's shard on every token. Expert parallelism gives whole experts to whole GPUs, which reads only the selected experts but requires an all-to-all to route tokens to them and back. The all-to-all is the cost, its backend is a configuration choice matched to the interconnect, and expert load imbalance is what actually limits the result.

Foundational

📐 AI Systems Design

The AI Infra Design Round PlaybookThe AI infrastructure design round is 45 to 60 minutes with one prompt (design a serving platform, a training scheduler, a 10k-GPU cluster, a fine-tuning service) and one interviewer whose job is to find the edge of what you know. The candidates who pass do the same things in the same order: pin the requirements and the numbers in the first five minutes, draw the reference shape, size it with a stated chain of arithmetic, pick two deep dives, and name the failure modes before being asked. This page gives that structure with a minute-by-minute plan, the numbers to bring in your head, and the mistakes that end the round early.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.Closest to this topicA mixture-of-experts model does an all-to-all twice per layer. What does that demand of the fabric, and what changes on a rack-scale system?Closest to this topicDesign inference for a 1T-parameter mixture-of-experts model: expert parallelism on an NVL72 rack, the attention cache, balance.

FEDITOR'S NOTE

Scored on separating total from active parameters in the first minute, on deriving the layout from the memory and communication budgets rather than naming a fashionable one, and on treating expert imbalance as a straggler problem with a measurable signal.

DISCUSSION · 0

No comments yet — be the first to share your approach.