How is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?

A global-array program lets the compiler derive communication from sharding choices. Work through a small matrix example, then separate that programming model from TPU topology, compilation and Pathways recovery.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

The TPU Software Stack: JAX, XLA, PyTorch and vLLM on TPUOn a GPU you run eager PyTorch and call tuned kernels. On a TPU a compiler sees the whole program, so shapes, sharding and compile time become your main tools. This maps JAX, MaxText, PyTorch/XLA, TorchTPU, Pallas and vLLM TPU onto the CUDA stack for training and serving.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

TPU Architecture and Systolic ArraysTPUs use matrix units and compiler-managed execution to run machine-learning workloads. Memory, shapes and inter-chip communication still determine how efficiently a model runs.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP reduce replicated training state by distributing it across workers. The memory saving comes with communication and temporary-buffer costs.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingConsistent hashing limits key movement when a shard set changes. It can preserve cache affinity, but it does not guarantee that cached data is present or that load is balanced.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackFine-tune a 70B on one 80 GB card. What do NF4 and double quantization actually buy, and where does DoRA change the arithmetic?Next in this trackDesign the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?Next in this trackYour team wants to replace a PPO-style RLHF pipeline with DPO. What leaves the cluster, and what does the training job look like afterwards?

FEDITOR'S NOTE

Explain named sharding with a concrete array layout. Separate compiler automation from correctness guarantees, hardware from framework choice, and runtime recovery from application-state recovery.

DISCUSSION · 0

No comments yet — be the first to share your approach.