AI Infra Interviews logo

TPU Model Co-Design: Shapes, Expert Traffic and Training Budgets

Learn how TPU model co-design connects local matrix shapes, ragged expert workloads and communication to quality at a fixed training budget, with worked examples and a practical study path.

BY RUIQI ZHOU · AIINFRAINTERVIEWS EDITORIAL · UPDATED NOVEMBER 18, 2026 · 4 MIN READ

PRACTICE THIS:Explore accelerator questions ·Study distributed training ·Kernels and the Machine course

TPU model co-design means choosing a model's architecture together with the way it runs on the accelerator. Matrix dimensions, expert routing, precision and device placement affect execution; those same choices can also change what the model learns. The useful objective is a model that meets quality and serving requirements within its training budget.

The open-source AI Hypercomputer TPU co-design guide connects these decisions across the stack. We reviewed its November 16 revision and built the examples below independently. They are calculations and experiment designs, not TPU benchmarks we ran.

Start with what you are allowed to change

An engineer deploying a released checkpoint can change supported layouts, batching and kernels while preserving its intended behavior. A team designing a new model can also change learned dimensions and architecture. That second set of choices needs training and quality evidence.

For example, doubling attention head dimension while halving head count can preserve the total projection width and logical cache size. It does not preserve the function of the attention heads. Their separate score distributions and learned representations have changed. Treating that proposal as a checkpoint reshape skips the central experiment.

The free TPU hardware comparison supplies system context. For a deployment migration, the TorchTPU guide provides a more appropriate starting point than changing the model's architecture.

Check the matrix after sharding

Suppose a projection has 6,144 output columns. It is divisible by 256, but the global width does not establish local tile fit. Under an illustrative kernel that rounds local columns to 256, eight equal shards receive 768 columns each and fit exactly. Sixteen shards receive 384 columns each and need 512 column positions.

The useful column fraction is 384/512 = 75%. This is one-axis tile accounting, not a measured utilization number or a prediction of a 25% slowdown. More devices also change arithmetic per shard, communication and memory fit. Inspect the actual compiled local shapes and compare complete work before choosing a layout.

The Premium tile-alignment question walks through the derivation, legal remedies and the distinction between padding an implementation and changing learned dimensions.

Let the valid data change inside a stable allocation

Expert batches are uneven. Four experts might receive 3, 0, 5 and 2 assignments on one step, then 1, 4, 0 and 5 on the next. Those changing counts need not force different buffer shapes.

A twelve-row allocation can hold either ten-assignment workload. Counts and prefix offsets identify the valid intervals; the allocation remains fixed. The kernel must preserve token identities and ignore invalid entries. Pallas's scalar-prefetch tutorial shows how small runtime metadata can drive data-dependent block indexing.

Storage pooling does not remove every partially filled execution tile. Nor does it solve overflow: thirteen assignments cannot fit twelve slots. A larger bucket or a split execution has a cost; silent dropping changes the computation. The Premium static buffers concept traces both the ordinary case and those boundaries.

Follow the proposed change to the evidence it needs
Comparison
Proposed changeImmediate effectEvidence still needed
Different shardingLocal shapes and communication changeMemory fit and complete-step timing
Fixed buffers with countsValid extents vary within a bounded allocationEmpty groups, tails, overflow and packing cost
Narrow expert activationsRouted payload can shrinkProjection cost and trained quality
Fewer, wider expertsRouting count and matrix width changeLoad distribution, task quality and serving behavior
Each row names a candidate intervention. None establishes an end-to-end speedup by itself.

Budget expert traffic separately from expert computation

LatentMoE introduces learned projections around the routed expert path. Tokens enter a smaller representation for expert computation, then return to the wider model representation. The Nemotron 3 report describes this architecture.

For a synthetic batch of 512 tokens routed to four experts each, full-width 4,096-value BF16 assignments carry 512×4×4,096×2 = 16 MiB of logical dispatch. Narrowing the routed width to 1,024 reduces that to 4 MiB. These numbers exclude locality, metadata and padding.

Doubling the active experts to eight spends some of the saving, bringing dispatch to 8 MiB. That may be a worthwhile quality trade, but it also changes computation and routing. The Premium LatentMoE concept separates those budgets and explains why expert compression does not automatically reduce attention-cache memory.

Another proposal is fewer, wider experts. Halving both expert count and top-K keeps average assignments per expert unchanged because both terms appear in B×K/E. Wider matrices can still help execution. The Premium expert-granularity question also shows why fewer touched experts need not mean fewer weight bytes.

Compare quality at the deadline

A throughput gain is useful when it improves the outcome you can afford. Compare against a tuned implementation of the original model, then isolate architecture changes enough to explain what caused the result. A combined change can identify a promising candidate without proving which individual choice helped.

Consider a hypothetical 24-hour experiment. A baseline processes 100,000 valid tokens per active second and spends one hour on compilation, evaluation and other overhead. It processes 8.280 billion tokens. A candidate at 120,000 tokens/s with two hours overhead processes 9.504 billion. Its active rate is 20% higher, while its token-count advantage is about 14.8%.

Evaluate quality at matched token counts and at the final deadline. The first comparison tests learning efficiency; the second answers the budget question. Neither token count supplies a quality score. Also measure serving at the required lengths, concurrency and latency targets: a training winner may be an expensive model to deploy.

A focused path through the material

Start with the public hardware and migration articles above. Then use the two Premium concepts to understand allocation and expert representations, and the questions to practice the calculations. The new Premium fixed-budget co-design lesson combines those ideas into an experiment memo and short drills inside Kernels and the Machine.

Keep the first experiment narrow enough to review. Record the intended change, the affected tensor, the predicted cost reduction, the quality criterion and the measurement that would make you reject it. That record turns an interesting optimization into a decision another engineer can check.

PRACTICE THIS

Turn it into offers. Work the real questions and concepts this maps to:

FAQ

It is the joint design of a model architecture and its execution on TPU hardware. It considers local matrix shapes, data movement, precision, memory and communication alongside training quality and serving requirements.

Discussion (4)

Ruiqi ZhouEditor

A useful review question is whether the proposed dimension change is legal for the checkpoint you actually own. The same shape table can describe a kernel choice or a new training project.

Morgan FosterEditor

Reserve an evaluation set before the architecture sweep. Selecting among 20 candidates on the same small benchmark can make the winning difference look more stable than it is.

Owen HartleyEditor

And budget the final confirmation run. Otherwise the selected candidate gets the last available accelerator hour and no repeat.

Deepa VenkateshEditor

The overflow path needs a test with all assignments landing on one owner. A pooled allocation can fit globally while one receiver still runs out of room.