← Distributed Training & Parallelism / 34

hardNewGoogleGoogle DeepMindAnthropic

A PyTorch team wants to train on TPUs. Do they rewrite in JAX with MaxText, use PyTorch/XLA, or wait for TorchTPU?

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Three routes with different costs: keep PyTorch through XLA, bet on the native backend, or adopt the JAX reference trainer. Choose from the model's custom code and the timeline, then prove parity before scaling.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧩 GPU & Accelerator Architecture🔒 Premium

The TPU Software Stack: JAX, XLA, PyTorch and vLLM on TPUOn a GPU you run eager PyTorch and call tuned kernels. On a TPU a compiler sees the whole program, so shapes, sharding and compile time become your main tools. This maps JAX, MaxText, PyTorch/XLA, TorchTPU, Pallas and vLLM TPU onto the CUDA stack for training and serving.

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.

🧩 GPU & Accelerator Architecture🔒 Premium

TPU Architecture and Systolic ArraysTPUs use matrix units and compiler-managed execution to run machine-learning workloads. Memory, shapes and inter-chip communication still determine how efficiently a model runs.

🧮 Napkin Math & Capacity🔒 Premium

Communication Volume EstimatesEstimate communication by naming the tensor, collective, group and synchronization schedule. Work through per-rank ring bytes, conditional sharding costs, activation exchanges and routing imbalance before converting traffic into exposed time.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicHow is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?Closest to this topicYou serve a model with vLLM on GPUs. What changes when you move the same model to TPUs?Closest to this topicYou have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?

DISCUSSION · 0

No comments yet — be the first to share your approach.