← Napkin Math, Cost & Capacity / 29

hardCoreWeaveTogether AIAWS

Rank the H100, MI300X and Trainium2 by cost per token for decode

Use bandwidth per dollar as a first comparison, then check feasible memory, supported execution and complete service cost before ranking accelerators.

Updated Sep 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Use bandwidth per dollar as a first comparison, then check feasible memory, supported execution and complete service cost before ranking accelerators.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

🧮 Napkin Math & Capacity🔒 Premium

Cost per Million TokensCost per million tokens connects fleet spending to useful output. Keep the time window, token definition and utilization assumptions consistent when comparing deployments.

🧩 GPU & Accelerator Architecture🔒 Premium

Trainium and InferentiaTrainium and Inferentia run workloads through AWS’s Neuron software stack. Evaluate memory, compiler support, interconnects and the cost of adapting the model.

🧩 GPU & Accelerator Architecture🔒 Premium

AMD Instinct and ROCmAMD Instinct GPUs offer substantial memory capacity and bandwidth. Workload performance also depends on ROCm, the model implementation and the kernels available for the chosen device.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA vendor claims 10,000 tokens per second per GPU. Sanity-check it.Closest to this topicHow many tokens per second can a 70B model generate for a single user on H100s?Closest to this topicHow many tokens per second can one B200 decode for a 70B model?

FEDITOR'S NOTE

Treat prices and efficiency factors as explicit examples; validate allocation units and comparable workloads before recommending hardware.

DISCUSSION · 0

No comments yet — be the first to share your approach.