Read the client clock before tuning the server
Separate response headers, first-token latency and completion. Work a four-request queue example and compare timely completions against all offered requests before choosing an admission limit.
12 MIN
TL;DR: An API can acknowledge a request long before the model starts it. Measure the first token at the client, preserve rejected requests in your results, and use queue limits to make an explicit latency-versus-admission decision.
Where you are. This free lesson bridges the local model benchmark and the Premium serving workshop. You can work through the example without running the earlier lab.
A fast model can have a slow front door
Imagine a model that needs 200 ms to produce its first token and 600 ms to finish a response. Those are hypothetical values for this exercise. Four readers ask at nearly the same time, but only one request can execute. The other three wait. Giving each waiting reader a quick HTTP acknowledgement does not make their answer arrive sooner.
HTTP is the request/response protocol between the client and server. The response begins with status and headers describing what follows. A streaming response then sends multiple pieces of a body over time. Here, the first useful piece is a token event: a record containing one predicted token ID. A header or a keepalive is not a token.
Time to first token, or TTFT, starts when this client begins sending the request and ends when it parses the first token event. That includes the local connection, request transfer, server work, waiting and response delivery. It still excludes drawing text on a screen. Put the boundary next to the number so someone else knows what was timed.
Follow four arrivals through one worker
Assume the worker processes one complete response at a time, each takes exactly 600 ms, no request is cancelled and transport overhead is zero. Number requests by execution order; simultaneous arrivals do not guarantee which reader gets which position.
Completion times are 600, 1,200, 1,800 and 2,400 ms. With a 1,500 ms completion deadline, two of four requests finish in time. Sanity check: four serial responses require 4 × 600 = 2,400 ms even though all arrived together. More waiting slots do not make the worker faster.
A rule that admits only one request would complete that one in 600 ms and reject the other three promptly. Its accepted-request latency looks better, but only one of four offered requests completes before the deadline. Rejecting late work can be the right choice when a caller can take a useful fallback. It is not automatically a throughput improvement.
Choose the denominator before the dashboard
| Measure | What belongs in the count | What it cannot establish |
|---|---|---|
| Accepted-request latency | Requests the server admitted | How many callers were refused |
| Deadline completions / offered requests | Every original request, including rejections | Whether the answer was correct |
| Token gaps | Time between successive token events at the client | Queue delay before the first token |
| Task-qualified goodput | Correct, permitted, timely answers under a stated workload | Quality when no acceptance test ran |
Keep transport completion separate from task correctness. A stream can deliver all its tokens perfectly and still contain an unsupported answer. The earlier small-model lab deliberately preserves that distinction. A fixed-token diagnostic is useful for comparing systems, but its extra predictions after a stop signal are not useful answers.
Decide what happens when there is no room
Admission is the decision to reserve capacity for a request. Backpressure tells an upstream caller that downstream capacity is limited. A bounded queue permits some waiting; rejection refuses work that cannot be admitted. Choose the waiting budget from the caller's deadline and observed service-time distribution, then measure whether the policy meets its objective.
A retry creates another attempt. If all rejected clients retry immediately, the server may receive a larger burst than the original one. A delay hint helps, but a real client needs a bounded retry policy, jitter and an overall deadline. The workshop deliberately makes zero automatic retries so the initial offered-work denominator stays visible.
The correct queue is not always empty. A short wait can absorb a brief burst without breaking the deadline. Reverse a no-wait policy when measured queued work reliably meets the objective and rejecting it prevents useful completions. Conversely, shrink a queue when waiting consumes the entire budget before execution begins.
Do this before moving on
Repeat the example with a 900 ms completion deadline. How many of the four complete in time with waiting, and how many with a one-request admission limit? Both policies deliver one timely completion, but the waiting policy still spends work finishing three late responses. Write the fallback or retry behavior that would make rejection useful to the caller. There is no universal winner without that product contract.
The following Premium workshops give you a real local API and a repeatable experiment. They keep model work serialized so you can see the queue directly; continuous batching and production fleet scheduling are later steps.
Go deeper
- Latency Metrics: TTFT, TPOT and Goodput defines the clock boundaries used in a serving report.
- Capacity and Backpressure connects waiting limits to overload and caller behavior.
- Designing for Latency SLOs turns a deadline into budgets across the request path.
- Failure modes of token streaming explains cancellation and incomplete delivery beyond the successful response.
Key takeaways
- Acknowledgement time and first-token time measure different events.
- Four serial 600 ms responses finish at 600, 1,200, 1,800 and 2,400 ms.
- Count rejected callers when reporting timely completions per offered request.
- Queue limits protect a budget; they do not create execution capacity.
Check yourself
Answer before you look. Recalling it is what makes it stick; recognising it does not.
1Four serial responses take 600 ms each. With a 1,500 ms completion deadline, how many finish in time?
2Why can accepted-request latency improve while fewer callers succeed?
Sign in to track which lessons you have finished.