slo

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

Derive a prompt-plus-output contract, calculate per-rank state capacity, and measure the concurrency that meets latency and overload targets.

Define a reproducible workload and service contract, account for every offered request, and choose a measured operating region with evidence and margin.

Turn a benchmark number into a bill that survives real utilization, then write the serving contract the gateway enforces and the evaluation the tuned configuration still owes.

A measured run shows where the time goes in a small batch job, then why every result parsed but none was deliverable. The answer turns on startup cost, co-tenant latency and a per-item contract.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.