hardNewCoreWeaveTogether AIAWS

Rank the H100, MI300X and Trainium2 by cost per token for decode

Decode is bandwidth-bound, so cost per token tracks dollars per terabyte per second, and on spec the MI300X wins by 2x. The table, the memory-capacity term that changes the replica shape, and the software-efficiency discount that decides whether the spec ranking survives a benchmark.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Cost per Million TokensThe unit every serving decision cashes out in. It is one formula: the fleet's dollars per second divided by the tokens per second it sustains, scaled to a million, with utilization in the denominator because idle replicas still cost money. This page derives it from a GPU price and a throughput estimate, works it at three batch sizes to show why batching is the main lever, separates prefill from decode pricing, and shows how the same fleet's cost per token moves by 5x between a quiet hour and a busy one.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

Trainium and InferentiaAWS's accelerators trade the GPU's general-purpose flexibility for a compiler-driven design with separate tensor, vector, scalar and GPSIMD engines, software-managed on-chip SRAM, and a proprietary NeuronLink fabric. Trainium2 delivers 667 dense bf16 TFLOPS with 96 GB at 2.9 TB/s, Trainium3 about the same bf16 with 2.5 PFLOPS of fp8 and 4.9 TB/s. The pitch is cost per FLOP; the price is a kernel ecosystem you may have to build yourself, which is exactly what the AWS loop probes.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

AMD Instinct and ROCmAMD's Instinct line competes on memory: 192 GB on the MI300X, 256 on the MI325X, 288 on the MI355X, with dense fp8 peaks at or above NVIDIA's at each generation. The catch is software: ROCm and HIP run most PyTorch and the main serving engines, but the attention, MoE and quantization kernels arrive on CUDA first and measured throughput has trailed the spec sheet. Interviewers want the numbers, the wavefront-of-64 difference, and a clear-eyed statement of when the memory argument wins.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA vendor claims 10,000 tokens per second per GPU. Sanity-check it.Closest to this topicHow many tokens per second can a 70B model generate for a single user on H100s?Closest to this topicHow many tokens per second can one B200 decode for a 70B model?

FEDITOR'S NOTE

The interviewer wants the candidate to pick bandwidth per dollar as the metric, build the table, and then discount it for measured efficiency and capacity. A ranking by TFLOPS is the wrong metric for decode and is the common miss.

DISCUSSION · 0

No comments yet — be the first to share your approach.