TL;DR: Cost per million = (fleet $/h ÷ 3,600) ÷ (tokens/s × utilization) × 1e6. Eight H100s at $2.50 is $20/h, $0.00556/s; at 2,000 output tokens/s fully utilized that is $2.78 per million; at 40% utilization it is $6.94. Throughput comes from the decode batch, so the same node ranges from $93 per million at batch 1 to under $2 at batch 64.
How to approach it
Ask three things: the GPU price, the operating batch (or a measured throughput), and the utilization the fleet actually runs at. Then say the formula with all three terms named. Compute the fleet's dollars per second first, since it is fixed, then get a throughput from the decode arithmetic and state the batch it assumes, then divide. Give the full-utilization number, then the realistic one, and close with the lever ranking: batch, utilization, bytes per token.
A strong answer
A typical situation: a published cost per million tokens is used to build a business case, and the real bill is two and a half times it. The published figure assumed a utilization nobody achieves.
The bill is the fleet's rental divided by the tokens it produced, and the tokens depend on how busy the fleet was and how many sequences it batched per step.
inputs: fleet = 8 × H100 at $2.50 per GPU-hour = $20 per hour
throughput = 2,000 output tokens/s at the operating batch (stated, or measured)
utilization = fraction of the hour spent generating
fleet $/s = 20 ÷ 3,600 = $0.005556 per second
cost per million tokens = fleet $/s ÷ (tokens/s × utilization) × 1e6
100% utilized: 0.005556 ÷ 2,000 × 1e6 = $2.78 per million
40% utilized: 0.005556 ÷ (2,000 × 0.4) × 1e6 = 0.005556 ÷ 800 × 1e6 = $6.94 per million
sanity: public list prices for a 70B-class open model are low single dollars per million output
tokens, which matches a high-batch, well-utilized fleet on below-list hardware; a private
deployment at 40% utilization landing at $7 is the same arithmetic with worse inputs.
Where does 2,000 tokens/s come from? The decode step on this node streams the weights once per step across the 8 cards and produces one token per sequence in the batch:
Llama 3.1 70B, fp8 weights (71 GB), TP8, batch 64, 4k average context, fp8 KV
weights per card = 71 ÷ 8 = 8.8 GB
KV per step (64 seqs) = 64 × 160 KB × 4,096 = 42 GB, per card 5.2 GB
bytes per card per step = 14 GB → 14e9 ÷ 3.35e12 = 4.2 ms
plus TP all-reduces and launch overhead ≈ 5 ms → step ≈ 9 ms
tokens/s ≈ 64 ÷ 0.009 ≈ 7,000 at batch 64 (a ceiling; 2,000 to 6,000 is what fleets sustain
once prefill, scheduling gaps and long tails are included)
The reason to show that chain is that it makes batch visible as the lever. The fleet price is fixed per hour, and tokens per second scale nearly linearly with batch until the compute ceiling, so cost per token falls in proportion:
| batch | tokens/s (approx.) | $/M at 60% utilization |
|---|---|---|
| 1 | 100 | $93 |
| 8 | 800 | $11.6 |
| 64 | 5,000 | $1.85 |
| 256 | 7,000 (near the compute ceiling) | $1.32 |
Everything that lets a replica run at a bigger batch for the same latency is a cost lever: continuous batching, an fp8 KV cache so more sequences fit, prefix caching so shared prompts are stored once, a larger-memory part. Everything that cuts bytes per step (fp8 or int4 weights) shifts the whole curve down.
Utilization is the second lever and it is set by traffic shape, not engineering. A chat fleet sized to hold its latency target at the noon peak idles overnight; 30 to 50% averaged over the day is normal, and it multiplies the cost by 2 to 3x. Autoscaling, multi-tenancy and scheduling batch work into the trough exist to raise that number.
Prefill is priced apart because it is compute-bound and fast: an 8k prompt on this node is 2 × 70.6e9 × 8,192 ≈ 1.16e15 FLOPs, about 0.24 s at 8 × 989 TFLOPS × 60%, so $0.0013 per prompt or about $0.16 per million input tokens. That is why input tokens list at a fraction of output tokens.
The reversal condition: at batch 1, a dedicated node for one latency-critical user, the cost is $93 per million, and whether that user's value justifies a dedicated node is a product decision rather than an engineering one. The Cost per Million Tokens concept carries the curve and the calculator. Capacity Planning and Utilization is where the utilization term comes from, and p99 TPOT is the constraint that stops you raising batch for a better number.
What interviewers probe next
- "Why is input cheaper than output?" Prefill is one compute-bound pass over the prompt; decode is one bandwidth-bound step per token. Per token, prefill is an order of magnitude cheaper.
- "What utilization should I assume?" 30 to 50% for an SLO-sized interactive fleet; 80%+ only for batch or offline workloads that can queue.
- "How would fp8 change the number?" Halves bytes per step, so at the same latency the batch can roughly double, and cost per token falls by close to half; see the fp8 question.
- "What if the GPUs are owned, not rented?" Replace $2.50 with the amortized hourly cost (capex over three years plus power, hosting and operations), which for an H100 fleet at high utilization is often $1.50 to $2.00, then run the same formula.
Common mistakes
- Dividing the GPU price by peak throughput with no utilization and no batch stated, producing a number nobody's fleet achieves.
- Reporting throughput as the batch-256 ceiling when the replica runs at batch 16 most of the day.
- Blending input and output tokens into one rate, which hides that a long-prompt workload is much cheaper per token than a long-output one.
- Forgetting that utilization is in the denominator, so a 40% fleet costs 2.5x, not 40% more.
Key takeaways
- $/M = ($/h ÷ 3,600) ÷ (tok/s × utilization) × 1e6; 8 H100s at $20/h and 2,000 tok/s: $2.78 full, $6.94 at 40%.
- Batch is the biggest lever: batch 1 to batch 64 is about 50x cheaper per token on the same node.
- Utilization of 30 to 50% is normal and multiplies cost by 2 to 3x.
- Prefill is compute-bound and about ten times cheaper per token; price it separately.