tpot

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Trace a 20,000-token prompt through a shared token budget, including the last partial chunk. Then choose a setting from measured first-token waits and streaming gaps.

Separate device throughput bounds from measured service curves, then choose a feasible operating point using latency, quality and goodput.

Define request success, first-output latency and streaming continuity for each service class. Count bad outcomes against an explicit budget, then use paired long and short windows to detect significant ongoing degradation.

Compare equivalent requests, inspect the replica’s software and device state, and reduce new traffic only after checking where that load will go.

Track request admission through a concrete page-reservation example, then connect scheduling, growth and stream cleanup to measured service limits.