TPU 8i Max vs 8tx vs 8t vs 8i: What Changed?
Compare the four TPU 8 systems, understand the dual-chip 8i Max and confidential 8tx designs, and size long-context memory without mistaking launch claims for benchmarks.
BY RUIQI ZHOU · AIINFRAINTERVIEWS EDITORIAL · UPDATED NOVEMBER 9, 2026 · 5 MIN READ
PRACTICE THIS:GPU and accelerator questions ·Inference and serving questions ·Calculate KV-cache memory
TPU 8 now has four named systems. 8t targets large-scale training; 8i targets latency-sensitive inference; 8tx adds a confidential-computing configuration; 8i Max combines two 8i chips in one package. The November additions change packaging, host and system design around the generation's existing silicon. That distinction matters when comparing memory, networking and software support.
Google announced 8i Max and 8tx on November 9, 2026. This article was checked on November 10 against the announcement and the 8t/8i technical description. Performance claims remain Google's reported results; no TPU 8 benchmark was run for this article.
Start with the resource your workload exhausts
An accelerator performs arithmetic, stores model state and exchanges data with other accelerators. High-bandwidth memory, or HBM, holds weights and attention state near the chip. SRAM is smaller on-chip storage. More of either can help, but they solve different parts of the problem.
| System | Stated emphasis | Architectural distinction |
|---|---|---|
| TPU 8t | Large training jobs | 3D torus; SparseCore; Axion hosts |
| TPU 8i | Sampling and serving | More SRAM; collectives engine; Boardfly |
| TPU 8tx | Confidential inference and smaller training | 8t silicon; AMD EPYC hosts; Boardfly |
| TPU 8i Max | Long context and rack density | Two 8i chips in one package |
The names describe emphasis. They do not prohibit running a different workload. For an existing service, identify whether it is waiting for memory, arithmetic, communication or a host-side operation before choosing a system. Prefill vs Decode explains why processing the prompt and generating the response can stress different resources.
What the 8i Max doubling means
The launch explicitly doubles compute, SRAM and HBM bandwidth relative to 8i. It does not say that every application becomes twice as fast. Nor should a reader obtain an HBM capacity figure by doubling an unrelated number in the April table. Use an exact configuration specification when it becomes available.
Consider an illustrative step that spends 8 ms moving model data and 2 ms on work unaffected by the extra bandwidth. Even if the data movement time halves perfectly, total time falls from 10 ms to 6 ms: a 1.67× speedup. If communication or scheduling consumes a larger fraction, the gain is smaller. This is a sensitivity calculation, not an 8i Max measurement.
Illustrative serialized step time, milliseconds
8 ms data movement + 2 ms other work
4 ms data movement + 2 ms other work
Increasing rack density can still be valuable when floor space limits capacity. Evaluate the rack's power, cooling and host capacity alongside accelerator count. A rack claim cannot be substituted for a measured cost per completed request.
Long context is a memory calculation
For a hypothetical full-attention model with 80 layers, 8 key/value heads, head dimension 128 and two-byte cache elements, each retained position uses:
KV bytes per position = 2 × 80 × 8 × 128 × 2
= 327,680 bytes = 320 KiB
At 1,000,000 positions = 327.68 GB ≈ 305.18 GiB per request
At 131,072 positions = 40 GiB per request
The leading factor of two counts keys and values. This example assumes full attention in every layer, no compression and no sharing between requests. Weights, workspaces, allocator padding and generated-token growth come on top. Hybrid attention or compressed state can change the result substantially.
Notice the scale: hundreds of gigabytes of cache do not fit inside hundreds of megabytes of SRAM. More SRAM can retain useful working tiles; it does not imply that an entire million-token conversation lives on-chip. Use the free KV-cache calculator, then verify the busiest device's allocation rather than summing a pod's advertised memory.
Turn the context limit into a concurrency budget
Google's 8i specification lists 288 GB of HBM per chip. Use that known quantity for an illustrative capacity check, keeping GB decimal throughout. Suppose exactly 70 billion BF16 parameters require 140 GB, and provisionally reserve 28.8 GB for everything other than weights and KV cache. The remaining cache allowance is 119.2 GB. That reserve is a planning assumption; a profile must replace it before deployment.
Using the full-attention model above, one request retaining 65,536 positions needs 21.475 GB of cache. Four independent requests need 85.899 GB; eight need 171.799 GB. The difference changes whether the proposed workload fits before any speed comparison begins.
| Allocation | Four requests | Eight requests |
|---|---|---|
| BF16 weights | 140 GB | 140 GB |
| Assumed non-cache reserve | 28.8 GB | 28.8 GB |
| KV at 65,536 positions each | 85.899 GB | 171.799 GB |
| Total against 288 GB | 254.699 GB: below capacity | 340.599 GB: exceeds capacity |
This is a necessary capacity check, not a serving recipe. The execution backend must support the model and device allocation, and its temporary peaks must fit the assumed reserve. A multi-device layout also needs a per-device check because weights, KV heads and workspaces may be unevenly placed.
Reserve for the largest retained history you allow, including generated tokens. A 65,536-token prompt with permission to generate another 8,192 tokens requires 73,728 retained positions under this full-attention assumption. At eight concurrent requests, that extra output allowance alone adds about 21.475 GB. Admitting by prompt length and hoping responses finish early turns a predictable budget into an out-of-memory incident.
More capacity can increase concurrency, context, or both. It does not establish the latency at that concurrency. After the fit check, measure first-token latency and token gaps at the same retained lengths; a configuration can fit in memory and still miss the service target.
What 8tx changes in a security review
The 8tx announcement describes protection of data and weights during use. An engineering review still needs the supported confidential mode, the trust boundary and the mechanism that proves the intended workload is running before keys are released. Those details decide whether an application meets its actual threat model.
For example, decrypting a prompt in an ordinary API gateway before forwarding it to protected accelerators leaves plaintext at that gateway. A secure accelerator cannot repair that earlier exposure. Likewise, prompts written to ordinary application logs remain outside the protected compute boundary. The Premium Confidential Inference reference develops that end-to-end design.
Google reports lower latency for 8tx relative to 8t, alongside a small confidential-computing penalty. Those are different comparisons. To measure the security overhead, compare supported protected and unprotected configurations with matched software and workload. Changing the network topology at the same time would mix two effects.
Choose a software path before reserving hardware
The hardware announcement and the installable software release have separate support boundaries. Start with TorchTPU migration for native PyTorch or the TPU software-stack concept for JAX and compiler responsibilities. For model-state movement between hosts, see TPU Sync.
TPU Agent Suite organizes development and profiling tasks. XLA Serving Engine addresses exported, short inference in a C++ runtime. Neither is a substitute for selecting a supported model-serving backend.
Before making a capacity commitment, write down the exact model revision, attention layout, required precision, engine release, parallel layout and target TPU configuration. Then require a correctness result, a cold-start measurement and a warm load sweep. A workload that cannot use a required operator or serving feature has no useful performance advantage, regardless of its peak specification.
For earlier hardware, the TPU generation comparison separates per-chip capacity from bandwidth and available slice shapes. That is the next step when an accessible older generation may fit the job today.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
Google announced TPU 8i Max on November 9, 2026. It packages two TPU 8i chips together and doubles compute, SRAM and HBM bandwidth relative to 8i. The announcement does not establish an equal increase in application throughput.
TPU 8tx combines 8t silicon with fifth-generation AMD EPYC hosts, hardware confidential-computing capabilities and a Boardfly network. It targets protected inference and smaller training workloads; 8t targets large-scale training.
No. Context capacity depends on the model architecture, cache precision, sharding and concurrent requests. A supported context length also needs a usable latency and quality result.
No. The reviewed launch sources do not provide a complete purchasable configuration, regional capacity and price matrix. Check the current Cloud TPU offering and software support matrix before planning a deployment.
Discussion (4)
A package-level comparison can hide the procurement unit. Ask whether the quote buys chips, a host, or an entire reserved slice before putting it into a cost spreadsheet.
Boardfly also changes the placement experiment. Keep the same parallel groups when you compare runs; otherwise the faster result may come from a better layout rather than the system alone.
And record the active topology. Maximum pod size is a poor substitute for the slice the job actually received.
For a code agent, I would put repeated-prefix and entirely new-repository requests in separate result rows. Their first-token costs can have very different causes.