TPU Generations Compared: v4, v5e, v5p, Trillium and Ironwood

Compare TPU generations by memory fit, bytes moved per step and communication layout. v5e and v6e offer relatively small per-chip memory; v5p and Ironwood provide more capacity and larger pod configurations. TPU 8 adds specialized training and inference designs, followed by 8tx and 8i Max system configurations. A larger generation number does not establish the lowest serving cost.

This November 10, 2026 comparison uses Google's generation documentation. It preserves the units published by each source because treating GiB as GB can change a fit decision. For the newest systems, read the separate TPU 8 comparison.

The per-chip comparison

BF16 is a 16-bit numerical format. A TFLOP/s is a trillion floating-point operations per second; these are peak compute figures. Bandwidth describes the rate at which data can move from HBM. Neither is a measured application rate.

The v5p documentation explains that its 95 GiB figure accounts for reserved memory from 96 GiB physical HBM. Other rows should not be assumed to use exactly the same reservation convention. Use the runtime's allocatable limit for an implementation.

For v5e, 800 GiB/s is about 859 GB/s, or 0.859 decimal TB/s. A chart that labels this as 0.8 TB/s changes the denominator. The following memory chart converts every row to decimal GB before plotting.

Capacity on one common scale

HBM per chip converted to decimal GB; not allocatable application memory

055110165220

v5e0–16

Published as 16 GB

v6e0–32

Published as 32 GB

v40–34.36

32 GiB converted

v5p0–102.01

95 GiB converted

Ironwood0–206.16

192 GiB converted

How many chips hold a 70B model?

Assume exactly 70 billion parameters stored at two bytes each. That is 140 GB, about 130.39 GiB, of weights. Ignore the cache and temporary tensors for this first lower bound.

Compare requirements before choosing a configuration

Comparison

Generation	Weight-only arithmetic	Minimum count by capacity
v5e	ceil(140 / 16)	9
v6e	ceil(140 / 32)	5
v5p	ceil(130.39 / 95)	2
Ironwood	ceil(130.39 / 192)	1

These counts are arithmetic, not valid purchase configurations. Nine chips of capacity does not mean a nine-chip slice is supported. A compiler or runtime may also expose a chip as multiple devices with separate local allocations. A model fitting the summed chip memory can still fail on one rank.

Now add a real request budget. Suppose the chosen layout leaves 80 GiB for cache and each growing request reserves 10 GiB. The payload allows at most eight such reservations before block padding or other overhead. A smaller prompt at admission does not permit more requests if every admitted request may grow to the reserved maximum. The free KV-cache sizing reference works through that distinction.

Training changes the memory comparison

The 140 GB estimate describes stored inference weights. Full fine-tuning must also keep gradients and optimizer state. For one explicit mixed-precision Adam recipe, assume BF16 weights and gradients, an FP32 master copy of the weights, and two FP32 optimizer moments. The moments track moving averages used by Adam's update.

An explicit 70B training-state recipe

Comparison

State	Bytes per parameter	70 billion parameters
BF16 weights	2	140 GB
BF16 gradients	2	140 GB
FP32 master weights	4	280 GB
Two FP32 Adam moments	8	560 GB

Perfectly partitioning all four categories across eight devices gives 140 GB of persistent state per device. Replicating them gives 1.12 TB on every replica. Sharding only the optimizer moments lands somewhere else again. That is why an inference fit table cannot select a training slice. The free Model Memory Footprint reference explains the allocation categories.

Even the fully partitioned estimate excludes saved activations, layer all-gathers, communication buffers and allocator overhead. A layer can temporarily assemble weights that are partitioned between steps, so peak memory can exceed the persistent share. Measure the busiest device during a complete optimizer update, including the first update when optimizer tensors may be created.

Different optimizers, gradient precision and master-weight policies change the 16-byte assumption. Adapter fine-tuning changes which parameters need gradients and optimizer state, while the frozen base weights still consume memory unless they are offloaded or quantized. Write down the actual recipe before comparing TPU generations; otherwise the comparison can be wrong by much more than the difference between two chips.

Why more FLOPS may barely change decode

For a simplified dense-model decode step, assume eight v6e chips evenly split the 140 GB weights and each reads its 17.5 GB share once. Dividing by the published 1,638 GB/s gives a 10.68 ms weight-read floor. Four Ironwood chips would each read 35 GB at 7,380 GB/s, giving 4.74 ms.

The configurations use different chip counts. This is a time-floor comparison, not a cost or equal-system benchmark. It excludes cache reads, communication, computation and scheduling. Real throughput also depends on how many requests share the weight read. Nevertheless, the calculation identifies which measurement to collect next: achieved memory throughput at the intended batch and context length.

The Roofline Model relates work to traffic. Using the table, v6e's BF16 ridge point is about 560 FLOPs per byte, while Ironwood's is about 313. Those thresholds do not assert that an entire model is memory-bound; apply them to an operation with consistent work and byte counts.

Topology determines the cost of splitting the model

v5e and v6e have 256-chip 2D pod topologies. v4, v5p and Ironwood use larger three-dimensional layouts. Slice dimensions, wraparound and supported allocation sizes determine which communication paths a job actually gets. The generation documentation is the source for those details.

Tensor parallelism exchanges partial results frequently. Expert parallelism moves tokens to the devices holding selected experts. A placement that performs well for one pattern can perform poorly for another, even with identical chip count. Draw the group that communicates on each layer and map it to the physical slice before attributing a slow collective to the chip.

The Premium TPU generations concept connects these layout choices to memory arithmetic. The accelerator architecture questions provide entry points for interview practice.

Check software support separately

The TorchTPU backend and a serving plugin have different support matrices. A generation supported by the backend may be experimental in the particular vLLM integration you need. Preserve the model, engine, backend and runtime versions together in a deployment record; a package name alone does not describe a reproducible environment.

Start with the TorchTPU migration guide, select an existing supported recipe, and run numerical checks before the performance sweep. Keep the current platform if the candidate misses a required model feature, quality threshold or latency target. Reconsider after a compatible release or a workload change, using the same acceptance criteria.