The Cost-per-Token Playbook

What the diagram shows

For 100,000 offered tasks, 90,000 accepted tasks share $4,500 of cost: $1,200 models, $300 tools, $2,400 human review and $600 shared service. The result is $0.05 per accepted task, not the $0.012 model-only cost per offered task.

Figure 1. Follow the full cost to the accepted outcome

Now suppose a cheaper model saves $400 in model charges but adds $900 of review work. With acceptance unchanged, total cost rises to $5,000 and cost per accepted case becomes about $0.0556, an 11.1% increase. If acceptance also changes, recompute the denominator. A model discount is evidence about one line on the invoice; the full workflow determines whether the product improved.

The FinOps Foundation distinguishes resource-efficiency units, such as cost per token, from business units, such as cost per case resolved. That distinction is useful here because it gives engineering and leadership a shared set of numbers without pretending they answer the same question. [1]

Keep a small scorecard

A business outcome need not be a sale. An internal assistant might save employee time, but the financial benefit requires evidence that the time was saved and used productively. A self-reported ten-minute saving is a different claim from ten minutes of avoidable paid work. Keep the distinction visible before converting time into a dollar return.

08 / CONNECT THE BUDGET TO CAPACITY

Buy capacity for the peak you intend to serve

A million requests per month sounds large until you divide it by time. Over a 30-day month, it averages about 0.386 requests/s. If most arrivals happen during a short daily window, the required serving rate can be many times higher. A monthly volume forecast pays for work; a peak forecast decides how much capacity must be ready at once.

State the interval behind “peak.” A five-minute average smooths bursts that may still overwhelm a two-second response target. Use arrival traces at the timescale that matters to the product, including launches, business hours and synchronized background jobs. Seasonal forecasts should identify which pattern is observed and which is a planning assumption.

Round the replicas up

Assume a tested replica serves 2 requests/s while meeting the workload's latency target. Plan to operate at 75% of that tested limit, allowing room for variation. Planned capacity is therefore 1.5 requests/s per serving replica. At a peak of 12 requests/s, the required serving count is ceil(12 / 1.5) = 8 replicas.

If the requirement includes losing any one replica without reducing that planned capacity, provision a ninth. With all nine available, capacity is 13.5 requests/s; with one unavailable, it is 12. The 75% fraction is explicit extra headroom. It is not a GPU-utilization measurement and should not be applied again if the input rate already included the same reserve.

What the diagram shows

A tested rate of 2 requests per second times a 75% planning fraction gives 1.5 requests per second per replica. A 12 requests per second peak needs ceil(12 / 1.5), or eight serving replicas. Add one spare: nine total. Normal planned capacity is 13.5 requests per second; after one replica fails, 12 remain. Spare capacity counts replicas, not zones.

Figure 8. Eight serving replicas and one spare

10 / CONNECT THE BUDGET TO CAPACITY

Find the crossover, then check the next capacity step

A team presents a neat graph: two lines cross, so self-hosting becomes cheaper above that volume. The arithmetic may be correct within the fleet's current capacity. The missing question is what happens when the next request requires another deployment.

Write costs in the same units and for the same accepted quality. For an illustrative month, suppose an API service costs $1,000 fixed plus $0.012 per offered task. A self-hosted service costs $9,000 fixed plus $0.002 per offered task, within a capacity band that supports at most one million such tasks and the specified arrival pattern.

The equality is $1,000 + $0.012V = $9,000 + $0.002V. Rearranging gives V = ($9,000 − $1,000) / ($0.012 − $0.002) = 800,000 tasks. At that volume, each option costs $10,600. Above it, within this band, the fleet is cheaper under the assumptions.

Make the capacity step visible

Now suppose the fleet's cost consists of $6,000 in common monthly costs, $3,000 per capacity block and $0.002 per task. One block supports up to one million tasks under this assumed traffic shape. Keep at least one block. The fleet cost becomes $6,000 + $3,000 × max(1, ceil(V / 1,000,000)) + $0.002V.

What the diagram shows

Illustrative monthly API cost is $1,000 plus $0.012 per task. Fleet cost is $6,000 plus $3,000 per million-task capacity block, rounded up with at least one block, plus $0.002 per task. The first crossover is 800,000 tasks. At one million, fleet cost is $11,000 versus $13,000 for the API. Just above one million the fleet jumps to $14,000, becoming more expensive until the second crossover at 1.1 million. Capacity is assumed for a fixed arrival pattern.

Figure 10. A new capacity block can reverse the saving