chunked prefill

Trace a 20,000-token prompt through a shared token budget, including the last partial chunk. Then choose a setting from measured first-token waits and streaming gaps.

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

A fourfold ratio is a symptom, not a diagnosis. Trace individual requests, separate waiting from execution and delivery, and test workload, scheduling, cache and capacity hypotheses against the service objective.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.