How is training on TPUs with JAX different from training on GPUs with PyTorch? What do you stop doing by hand?

On a TPU pod you write one program, annotate how each array is sharded across a named mesh, and the compiler inserts and schedules every collective. What that removes from the engineer's job, what the torus changes about layouts, where Pathways fits, and what a compiler-owned schedule costs.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

TPU Architecture and Systolic ArraysA TPU is a matrix unit first and a processor second: a systolic array that streams activations through a grid of multiply-accumulate cells holding stationary weights, fed by a compiler rather than a warp scheduler, and scaled out over a dedicated inter-chip interconnect into pods of thousands. The roofline thinking transfers from GPUs unchanged; the workflow does not, which is what the Google loop tests.

Core

🕸️ Distributed TrainingSign in

ZeRO and FSDPZeRO and FSDP keep data parallelism's simple programming model but shard the optimizer state, gradients and parameters across ranks, cutting per-GPU memory from 16 bytes per parameter toward 16/N. The price is 1.5x DDP's communication and a dependence on tokens per GPU that decides when sharding stops paying and tensor parallelism takes over.

Foundational

💻 Coding for Infra

Consistent Hashing and ShardingSplitting work across N servers with a modulo of N moves almost everything when N changes, which for a cache means throwing away almost all of it. Consistent hashing places servers and keys on a ring so adding or removing one moves only its share, and virtual nodes fix the imbalance a small ring otherwise has. In LLM serving the same structure routes requests by prompt prefix so a conversation reaches the replica already holding its cache.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackFine-tune a 70B on one 80 GB card. What do NF4 and double quantization actually buy, and where does DoRA change the arithmetic?Next in this trackDesign the infrastructure for RLHF on a 70B: where do rollouts and the learner run, and how do weights move between them every step?Next in this trackYour team wants to replace a PPO-style RLHF pipeline with DPO. What leaves the cluster, and what does the training job look like afterwards?

FEDITOR'S NOTE

Scored on explaining SPMD sharding annotations concretely, on knowing that XLA emits and overlaps the collectives, on the torus versus NVSwitch difference, and on the honest costs: compile time, static shapes, and less control.

DISCUSSION · 0

No comments yet — be the first to share your approach.