Advanced

TPU Generations: v4 to TPU 8

TPU generations differ along three axes that decide a deployment: memory per chip, the ratio of compute to memory bandwidth, and how chips connect. Small-memory 2D-torus parts, large 3D-torus pods and the training/inference split in TPU 8 each fit different workloads.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

TPU Architecture and Systolic Arrays GPU Generations: A100 to Blackwell Roofline Model Topology-Aware Communication KV Cache Sizing TCO: Buy vs Rent

LESSONS THAT TEACH THIS

Kernels and the MachineOn a compiled accelerator, the program is the unit, not the kernel22 min→Kernels and the MachineServe on a compiled accelerator: choose buckets, budget warmup, size per device26 min→

PRACTICE THIS IN REAL QUESTIONS

GPU & Accelerator ArchitectureYou have trained on GPUs. What transfers to training on TPUs, and what do you have to relearn?→Hardware, Cabling & Cluster Build-OutYour fleet is decode-heavy. Is a B300 worth 1.4 times a B200's power for 1.6 times the memory?→GPU & Accelerator ArchitectureDescribe the GPU memory hierarchy. Where can a byte live on an H100, and what does each level cost?→Hardware, Cabling & Cluster Build-OutYou are building a fine-tuning service for customer models under 30B. RTX PRO 6000 or H100?→GPU & Accelerator ArchitectureExplain what HBM is and why memory bandwidth, not compute, is the wall for LLM inference.→GPU & Accelerator ArchitectureExplain how a systolic array computes a matrix multiply. Why is it efficient, and why does it want big square tiles?→