ttft

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Trace a 20,000-token prompt through a shared token budget, including the last partial chunk. Then choose a setting from measured first-token waits and streaming gaps.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

A fourfold ratio is a symptom, not a diagnosis. Trace individual requests, separate waiting from execution and delivery, and test workload, scheduling, cache and capacity hypotheses against the service objective.

Work through projection and attention FLOPs for an 8,192-token prompt. Separate a peak-based lower bound, an assumed-rate estimate and observed TTFT.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.