TL;DR: Divide the cost of a declared interval by its useful output tokens, then multiply by one million. In a hypothetical $20 hour at 2,000 output tokens per active second, full activity costs $2.78 per million; 40% activity costs $6.94. If the measured token rate already includes idle time, do not multiply it by utilization again.
How to approach it
Define the cost boundary, token denominator and time interval first. Use measured complete-window counters when available. Only decompose throughput into an active rate and activity fraction when those definitions are explicit. Close with latency, quality and failure constraints on any cheaper configuration.
A strong answer
A benchmark runs a busy replica for a minute; the bill covers a day with quiet hours and failed requests. Dividing the daily cost by the benchmark's maximum output rate describes neither interval. Cost per Million Tokens and Capacity Planning and Utilization separate the accounting choices.
Take assumed, not quoted market, prices: eight GPUs at $2.50 per GPU-hour, or $20 per fleet-hour. Suppose the replica produces 2,000 useful output tokens per active second, and the entire fleet is active for 40% of the hour.
interval cost = 8 × $2.50 × 1 hour = $20
active seconds = 3,600 × 0.4 = 1,440
output = 2,000 × 1,440 = 2,880,000 tokens
cost per million = 20/2,880,000 × 1e6 = $6.9444
whole-hour rate = 2,880,000/3,600 = 800 tokens/s
sanity: 20/(800×3,600) × 1e6 gives the same $6.9444.
At full activity the output becomes 7.2 million and cost becomes $2.7778 per million. At 20% it is $13.8889. The inverse relationship follows only while the assumed active rate and hourly cost stay fixed. Real autoscaling changes capacity and cost; batch composition changes active throughput.
Here, activity means the fraction of the hour during which the fleet produces the assumed 2,000 useful tokens/s. It is not a GPU activity counter, tensor-pipeline activity, MFU or memory occupancy. Those measurements answer different questions and cannot be substituted into this denominator. If the fleet size changes during the hour, add the actual costs and useful tokens over that same interval rather than averaging replica percentages.
| Denominator | What the result means |
|---|---|
| All generated output | Includes wasted or rejected generation if counted |
| Delivered output | Requires a defined treatment of disconnects and retries |
| Output from SLO-qualified requests | Includes latency/failure quality in the service result |
| Input plus output | A workload-specific blended metric, not output-token cost |
The serving benchmark's definitions matter. vLLM's benchmark interface exposes workload and service measurements; record the model, numerical format, prompt/output distribution, concurrency and timing boundary alongside the result. Do not derive a measured 2,000 tokens/s from an ideal HBM floor plus invented overhead.
Suppose the same $20 hour generates 2.88 million output tokens but only 2.4 million belong to requests meeting the chosen service contract. The qualified-output cost is 20/2.4 = $8.3333 per million. The difference is useful: a configuration can lower raw-token cost while increasing missed deadlines or failures. Keep the criterion stable across comparisons.
GPU rent is one cost boundary. A complete service view may add host compute, network, storage, orchestration and operational costs. Owned hardware requires a stated allocation policy for capital and recurring costs; there is no universal hourly ownership price. Likewise, output-only cost may allocate all prefill expense to output. If separating input and output costs, explain the attribution method and avoid charging the same shared cost twice.
I would compare supported batch and precision settings at matching quality and latency objectives. Keep a change only if it improves the relevant cost denominator without violating the service contract. Higher concurrency can worsen token gaps; quantization can add overhead or fail a quality test; caching helps only the workload with compatible reuse. None has a universal cost multiplier.
What interviewers probe next
- “Why not use peak throughput?” It is a capacity experiment, not a complete-window production denominator. It can inform a scenario if labeled as an assumption.
- “Do failed requests disappear from cost?” Their resource cost remains. Decide explicitly whether their tokens belong in the useful-output denominator.
- “Is input always ten times cheaper?” No. Relative cost depends on prompt lengths, caching, hardware, batching and the allocation policy.
Common mistakes
- Applying an activity discount to an already whole-window rate.
- Treating synthetic prices or throughput as published market data.
- Comparing token costs at different quality or latency targets.
Key takeaways
- The hypothetical 40%-active hour produces 2.88M tokens for $20: $6.94/M.
- A measured 800-token/s whole-hour rate already includes that idle time.
- Qualified throughput can change the cost decision even when raw throughput improves.