TPU Model Co-Design: Shapes, Expert Traffic and Training Budgets
Learn how TPU model co-design connects local matrix shapes, ragged expert workloads and communication to quality at a fixed training budget, with worked examples and a practical study path.
BY RUIQI ZHOU · AIINFRAINTERVIEWS EDITORIAL · UPDATED NOVEMBER 18, 2026 · 4 MIN READ
PRACTICE THIS:Explore accelerator questions ·Study distributed training ·Kernels and the Machine course
TPU model co-design means choosing a model's architecture together with the way it runs on the accelerator. Matrix dimensions, expert routing, precision and device placement affect execution; those same choices can also change what the model learns. The useful objective is a model that meets quality and serving requirements within its training budget.
The open-source AI Hypercomputer TPU co-design guide connects these decisions across the stack. We reviewed its November 16 revision and built the examples below independently. They are calculations and experiment designs, not TPU benchmarks we ran.
Start with what you are allowed to change
An engineer deploying a released checkpoint can change supported layouts, batching and kernels while preserving its intended behavior. A team designing a new model can also change learned dimensions and architecture. That second set of choices needs training and quality evidence.
For example, doubling attention head dimension while halving head count can preserve the total projection width and logical cache size. It does not preserve the function of the attention heads. Their separate score distributions and learned representations have changed. Treating that proposal as a checkpoint reshape skips the central experiment.
The free TPU hardware comparison supplies system context. For a deployment migration, the TorchTPU guide provides a more appropriate starting point than changing the model's architecture.
Check the matrix after sharding
Suppose a projection has 6,144 output columns. It is divisible by 256, but the global width does not establish local tile fit. Under an illustrative kernel that rounds local columns to 256, eight equal shards receive 768 columns each and fit exactly. Sixteen shards receive 384 columns each and need 512 column positions.
The useful column fraction is 384/512 = 75%. This is one-axis tile accounting, not a measured utilization number or a prediction of a 25% slowdown. More devices also change arithmetic per shard, communication and memory fit. Inspect the actual compiled local shapes and compare complete work before choosing a layout.
The Premium tile-alignment question walks through the derivation, legal remedies and the distinction between padding an implementation and changing learned dimensions.
Let the valid data change inside a stable allocation
Expert batches are uneven. Four experts might receive 3, 0, 5 and 2 assignments on one step, then 1, 4, 0 and 5 on the next. Those changing counts need not force different buffer shapes.
A twelve-row allocation can hold either ten-assignment workload. Counts and prefix offsets identify the valid intervals; the allocation remains fixed. The kernel must preserve token identities and ignore invalid entries. Pallas's scalar-prefetch tutorial shows how small runtime metadata can drive data-dependent block indexing.
Storage pooling does not remove every partially filled execution tile. Nor does it solve overflow: thirteen assignments cannot fit twelve slots. A larger bucket or a split execution has a cost; silent dropping changes the computation. The Premium static buffers concept traces both the ordinary case and those boundaries.
| Proposed change | Immediate effect | Evidence still needed |
|---|---|---|
| Different sharding | Local shapes and communication change | Memory fit and complete-step timing |
| Fixed buffers with counts | Valid extents vary within a bounded allocation | Empty groups, tails, overflow and packing cost |
| Narrow expert activations | Routed payload can shrink | Projection cost and trained quality |
| Fewer, wider experts | Routing count and matrix width change | Load distribution, task quality and serving behavior |
Budget expert traffic separately from expert computation
LatentMoE introduces learned projections around the routed expert path. Tokens enter a smaller representation for expert computation, then return to the wider model representation. The Nemotron 3 report describes this architecture.
For a synthetic batch of 512 tokens routed to four experts each, full-width 4,096-value BF16 assignments carry 512×4×4,096×2 = 16 MiB of logical dispatch. Narrowing the routed width to 1,024 reduces that to 4 MiB. These numbers exclude locality, metadata and padding.
Doubling the active experts to eight spends some of the saving, bringing dispatch to 8 MiB. That may be a worthwhile quality trade, but it also changes computation and routing. The Premium LatentMoE concept separates those budgets and explains why expert compression does not automatically reduce attention-cache memory.
Another proposal is fewer, wider experts. Halving both expert count and top-K keeps average assignments per expert unchanged because both terms appear in B×K/E. Wider matrices can still help execution. The Premium expert-granularity question also shows why fewer touched experts need not mean fewer weight bytes.
Compare quality at the deadline
A throughput gain is useful when it improves the outcome you can afford. Compare against a tuned implementation of the original model, then isolate architecture changes enough to explain what caused the result. A combined change can identify a promising candidate without proving which individual choice helped.
Consider a hypothetical 24-hour experiment. A baseline processes 100,000 valid tokens per active second and spends one hour on compilation, evaluation and other overhead. It processes 8.280 billion tokens. A candidate at 120,000 tokens/s with two hours overhead processes 9.504 billion. Its active rate is 20% higher, while its token-count advantage is about 14.8%.
Evaluate quality at matched token counts and at the final deadline. The first comparison tests learning efficiency; the second answers the budget question. Neither token count supplies a quality score. Also measure serving at the required lengths, concurrency and latency targets: a training winner may be an expensive model to deploy.
A focused path through the material
Start with the public hardware and migration articles above. Then use the two Premium concepts to understand allocation and expert representations, and the questions to practice the calculations. The new Premium fixed-budget co-design lesson combines those ideas into an experiment memo and short drills inside Kernels and the Machine.
Keep the first experiment narrow enough to review. Record the intended change, the affected tensor, the predicted cost reduction, the quality criterion and the measurement that would make you reject it. That record turns an interesting optimization into a decision another engineer can check.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
It is the joint design of a model architecture and its execution on TPU hardware. It considers local matrix shapes, data movement, precision, memory and communication alongside training quality and serving requirements.
No. Supported layouts and equivalent kernel changes may preserve the checkpoint's function. Changing head count, expert width or learned projection layers changes the model and requires an appropriate training and quality-evaluation plan.
No. Fixed-capacity buffers can carry changing valid regions described by runtime counts and offsets. The implementation still needs bounds checks, correct masking, preserved token identities and an explicit overflow policy.
Only if the other terms support that conclusion. Mean assignments per expert are B times K divided by E. Halving both expert count E and top-K leaves that average unchanged at fixed batch B.
Discussion (4)
A useful review question is whether the proposed dimension change is legal for the checkpoint you actually own. The same shape table can describe a kernel choice or a new training project.
Reserve an evaluation set before the architecture sweep. Selecting among 20 candidates on the same small benchmark can make the winning difference look more stable than it is.
And budget the final confirmation run. Otherwise the selected candidate gets the last available accelerator hour and no repeat.
The overflow path needs a test with all assignments landing on one owner. A pooled allocation can fit globally while one receiver still runs out of room.
