hardNewAWS

Would you move a cost-sensitive training and serving fleet from H100 to Trainium2? What do you gain, and what do you have to plan for?

A Trn2 instance carries 16 Trainium2 chips with 1.5 TB of HBM and more dense bf16 FLOPS than an 8-GPU H100 node. Whether the cheaper FLOPS reach your workload depends on the Neuron compiler, the kernels you do not have yet, and an MFU number you must measure rather than assume.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Trainium and InferentiaAWS's accelerators trade the GPU's general-purpose flexibility for a compiler-driven design with separate tensor, vector, scalar and GPSIMD engines, software-managed on-chip SRAM, and a proprietary NeuronLink fabric. Trainium2 delivers 667 dense bf16 TFLOPS with 96 GB at 2.9 TB/s, Trainium3 about the same bf16 with 2.5 PFLOPS of fp8 and 4.9 TB/s. The pitch is cost per FLOP; the price is a kernel ecosystem you may have to build yourself, which is exactly what the AWS loop probes.

Foundational

🧭 Ownership & Judgment

Talking About Cost and Capacity with LeadershipInfrastructure engineers are asked to justify large numbers to people who do not share their vocabulary, and the conversations go wrong in predictable ways: a technical objection with no alternative, a forecast with no assumptions, or a cost quoted in a unit the listener cannot act on. What works is a small number of costed options, a stated recommendation, the decision needed by a date, and every figure expressed in whatever the listener actually controls.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Capacity Planning and UtilizationCapacity planning for GPUs is deciding how many to have next quarter given that they cost money whether busy or not, that demand arrives in bursts, and that a queue near saturation produces waits that grow without bound. This page works the planning arithmetic for a serving fleet (peak demand, headroom, the p99 penalty of running hot) and a training platform (job mix, queue wait, the value of a shared pool), and gives the queueing intuition that makes 70% look full. The number that decides everything is utilization, and it has a ceiling set by latency, not by hardware.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

TCO: Buy vs RentWhether to buy GPUs or rent them is a utilization question dressed as a finance question. An owned H100 costs a few tens of thousands of dollars up front and a known amount per hour in power, cooling, space and operations; a rented one costs a few dollars per hour and nothing when idle. The break-even is the utilization at which the owned hourly cost, amortized over the hardware's useful life, equals the rental rate. This page builds the owned cost from parts, works the break-even, and adds the terms the simple model leaves out: depreciation risk, reserved discounts, and the price of idle capacity.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackThe MI300X has 192 GB per card. When does AMD actually win against an H100 for LLM serving, and what would you check before betting on it?Next in this trackWhy are Groq and Cerebras so fast at batch 1, and what does that speed cost at scale?Next in this trackMIG versus MPS: what isolation does each give you when sharing a GPU, and which would you pick for a multi-tenant inference node?

FEDITOR'S NOTE

Scored on a spec-sheet comparison done per instance rather than per chip, a break-even formula that includes achieved MFU and engineering cost, and a concrete list of what the compiler-first toolchain requires the team to plan for.

DISCUSSION · 0

No comments yet — be the first to share your approach.