Cost per million tokens calculator

Q: Why does cost per token fall so steeply with batch size?

Because decode at small batch is memory-bound: the weights are read once per step whether that step serves one sequence or sixty-four. Until the batch reaches the ridge point, tokens per second scales almost linearly with batch and cost per token falls in proportion.

Q: Why is utilization in the formula?

A replica costs the same per hour whether or not it is generating. Bursty traffic leaves it idle a large fraction of the time, and that idle time is spread over the tokens you did serve. Autoscaling and multi-tenancy exist to push this fraction up.

Derive the cost of a million output tokens from GPU price, replica size, batch and context, using bandwidth-bound and compute-bound decode throughput estimates, or plug in your measured tokens per second.

Deployment

Model

Weights dtypeAccelerator

GPUs per replicatensor-parallel degree$ per GPU-hourUSD

Traffic

Decode batch: 64 sequences64Average context per sequence: 4,096 tokens4kUtilization: 60%60%share of the hour the replica is actually generating; bursty traffic sits well under 100%Measured tokens/s per replica (optional)tok/soverrides the estimate below when you have a benchmark

Cost per million output tokens

$1.23

4 × H100 80GB SXM at $10.00/h, 60% utilized

Estimated throughput

4ktok/s

min(bandwidth bound 4k, compute bound 17k)

Decode regime

memory-bound

intensity ≈ 64 FLOP/B vs ridge 295

Memory at this batch

227 GB

fits in the replica at 90%

How this was computed

Bytes per step: 141 GB weights + 64 × 1.3 GB KV = 227 GB
Steps per second: 13.4 TB/s ÷ bytes per step = 59.0
Bandwidth-bound tok/s: steps/s × 64 = 4k
Compute-bound tok/s: 4 × 989e12 × 60% ÷ (2 × 70.6B) = 17k
$ per M tokens: ($10.00 ÷ 3600) ÷ (4k × 60%) × 1e6 = $1.226

Prefill is charged separately in practice and is compute-bound, so input tokens cost less per token at long prompts than this decode figure. Weights in 2 B/param; KV kept in bf16.

What the interviewer is actually asking

Neoclouds and serving startups ask for the dollars-per-token math because it is where every serving decision cashes out: batch size, quantization, tensor-parallel degree and utilization all change the number by multiples. The interviewer wants the mechanism, bandwidth-bound decode turning compute-bound as batch grows, not a price they can look up.

Questions people ask

How do you estimate tokens per second for decode?▲

Each decode step streams the active weights once plus every sequence's KV cache, and produces one token per sequence. Divide aggregate memory bandwidth by bytes per step for steps per second, multiply by batch for tokens per second, and cap at the compute-bound rate of peak FLOPS divided by two FLOPs per active parameter.

Why does cost per token fall so steeply with batch size?▼

Why is utilization in the formula?▼