A mixture-of-experts layer is 256 small matmuls. Why is a loop over cuBLAS calls slow, and what does a grouped GEMM do instead?

Separate per-expert submission, small-grid scheduling and partially filled tiles. A grouped kernel can share work across experts, but routing, packing and output combination still determine the layer’s cost.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

Roofline calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEExpert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.

Foundational

🧮 Open Weights & Serving Engines

Capacity Planning for Open-Weights FleetsMatch forecast traffic to a measured replica operating envelope, then check failure domains, rollout capacity, readiness time and cost. Output tokens alone omit prompt work and live state; total parameter count alone does not determine a replica's throughput.

Foundational

🧮 Open Weights & Serving Engines

Reading config.json to Size a Model You Have Never RunRead a model configuration as an architecture description, then combine it with tensor metadata and engine allocation rules. Learn which fields determine attention state, why mixed precision needs tensor-level accounting, and how to turn a first estimate into a testable per-GPU capacity plan.

Foundational

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsCompare tensor and expert placement without confusing either with sparsity. Derive logical dispatch traffic and redundant-expert memory, then choose backends and balancing from per-rank measurements.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackSelect the top k logits from a 128,000-token vocabulary on the GPU. What shape does the kernel take and why not just sort?Next in this trackYou added three local variables to a working kernel and it got 30 percent slower. Explain what happened and how you would confirm it.Next in this trackA training step runs at 20 percent model FLOPs utilization. Profile it and find where the missing time goes.

FEDITOR'S NOTE

Count token-expert assignments and tile capacity separately, explain what grouping changes, and include permutation and weighted combination in the measured layer. Avoid treating a routing change as a semantics-free kernel optimization.

DISCUSSION · 0

No comments yet — be the first to share your approach.