mediumNVIDIACoreWeave

How many tokens per second can one B200 decode for a 70B model?

Count weight and cache traffic on a specified B200 configuration, check finite capacity, and separate aggregate throughput from per-user pace. Worked bounds guide a benchmark rather than replace it.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Count weight and cache traffic on a specified B200 configuration, check finite capacity, and separate aggregate throughput from per-user pace. Worked bounds guide a benchmark rather than replace it.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

GPU Generations: A100 to BlackwellGPU generations change memory, arithmetic support and interconnect capacity at different rates. Compare the specification that limits your workload rather than the generation name.

Foundational

🖧 Hardware & Cluster Build-Out

Accelerator Selection: H100 to B300 and RTX PRO 6000Choose an accelerator by checking memory capacity, memory bandwidth and compute throughput separately. Each limits a different part of the workload.

Core

🧮 Napkin Math & CapacitySign in

Arithmetic Intensity by OperationCount the FLOPs and memory transfers of a projection, attention and a residual add. Learn where the batch-size shortcut works, where it breaks, and which hardware roof applies.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat storage throughput does a 16k-GPU training cluster need?Next in this trackTraffic peaks at three times the daily average. Capacity-plan the serving fleet.Next in this trackWhat does one training token cost?

FEDITOR'S NOTE

Separate a bandwidth-only ceiling from measured throughput, verify memory headroom, and report aggregate output and per-user pace with explicit assumptions.

DISCUSSION · 0

No comments yet — be the first to share your approach.