← Napkin Math, Cost & Capacity / 14

hardDeepSeekMistralGoogle

A MoE model and a dense model reach the same quality. How do their GPU counts compare for serving and for training?

Separate resident parameters, active work and the union of experts touched by a batch. Compare toy dense and MoE budgets without promising a fixed speedup.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Separate resident parameters, active work and the union of experts touched by a batch. Compare toy dense and MoE budgets without promising a fixed speedup.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEExpert parallelism places different MoE experts on different devices. Routing tokens between them adds communication and makes uneven expert demand a performance concern.

🧮 Napkin Math & CapacitySign in

GPU-Hours and Time to TrainConvert a stated work budget into elapsed time and GPU-hours. Vary utilization, invert for a deadline, and separate productive time from calendar delays without counting them twice.

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

🧮 Napkin Math & Capacity

Model Memory FootprintSeparate weight payload, growing request state and training allocations. Work through exact units and per-rank budgets, then check the live peak.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackEstimate how long it takes to write a checkpoint for a 405B training run Next in this trackWhat is the power budget for a cluster of 1,024 H100s?Next in this trackShould we buy or rent 1,000 H100s for the next two years?

FEDITOR'S NOTE

Trace the example, state its assumptions, and distinguish the estimate from measured behavior.

DISCUSSION · 0

No comments yet — be the first to share your approach.