Measure useful answers before renting more compute

TL;DR: Judge a faster model by the answers people can use. Compare the same cases, count failures and late responses, and choose evaluation thresholds before looking at the final test results.

Where you are. You can identify which release a service is running. Now you will decide whether its answers are good enough to approve that release.

A successful response can contain an unusable answer

Our documentation service returns text when someone asks a question. To decide whether that response is useful, we also need to check that it answers the question, cites current evidence the caller may access, and arrives in time. Write those acceptance rules before collecting results so you can judge every candidate by the same standard.

In this exercise, an acceptable answer addresses the question, is supported by its cited passage and arrives before the stated deadline. When the documents cannot support an answer, the service should say so; this is called abstaining. Record quality, authorization and timing separately even though the final acceptance rule requires all three. That way, a failed result tells you what to investigate.

These hypothetical counts account for every original request: 72 acceptable-on-time, 9 acceptable-late, 9 wrong and 10 unfinished make 100. Completed-answer quality counts only the requests that finished: 81/90 = 90%. Acceptable-on-time service output counts every original request: 72/100 = 72%. Both are useful, but substituting one for the other hides the cost of failures and delay.

Valid JSON can ask for the wrong thing

Suppose a user asks, “Check my quota in east.” A tool call is the model's proposal to invoke an application operation with arguments. It does not grant permission or execute the operation by itself. Our read-only quota tool accepts a region field with either east or west.

The model proposes {"region":"west"}. JSON is the text format; a schema is the rule describing allowed fields and values. This response passes both checks and still misses the request.

Constrained generation limits which output shapes or values the model can produce. It can keep the proposed region within east or west, but both are allowed, so it cannot establish that the model chose the one the user requested. Permissions need a separate check too: the application should use the signed-in caller's account, rather than trusting an account identifier supplied by the model.

Wait for the response's documented completion signal before treating its arguments as final. A stream fragment or an interrupted response can look like a complete object. Record that failure separately from malformed JSON and wrong task meaning. In an open-ended task, deciding whether the meaning is correct may require reference evidence or a reviewed rubric; a string comparison is enough only for a fixture with a known expected value.

Add four columns to the evaluation record: completed normally, valid format, permitted operation, correct task result. A single “valid JSON” score would conceal the failure shown here.

Compare the same cases

Give each test case a stable identifier and save its input, reference evidence, permission scope and expected behavior. Run the current release and the candidate on those same cases, then store their outputs and failures with the release identities. If a request needs several attempts, retain each attempt for diagnosis while counting the original request once in the user-facing result.

Suppose 100 paired cases produce this table, where success means passing the full acceptance rule:

Baseline success is 72%, from the first two rows; candidate success is 78%, from the first and third rows. The candidate gained ten cases and lost four. Inspect those four regressions before deciding the six-point aggregate increase is worthwhile. A failure on a permission-sensitive question can matter more than several minor formatting improvements. This small example does not establish statistical significance.

Learn the metric that matches the decision

Choose a metric that answers the decision you need to make. If a model routes requests into two groups, precision and recall describe false alarms and missed cases. A retrieval test asks whether the needed source was found. An answer-quality test asks whether the response is supported and correct for the task. Text similarity alone cannot answer all of these questions.

A threshold is the score at which your system takes an action. Choose it on validation data, weighing the consequences of each kind of error, and save a separate test set for the final check. Keep closely related documents or conversations in the same split. Otherwise, near-duplicate examples can make the test look easier than the unseen work the service will receive.

These choices sit around the model: how inputs are represented, how outputs are interpreted and how success is judged. A deep-learning framework supplies the computation, but you still have to choose suitable labels and evaluation rules. A notebook is useful for exploring examples; a repeatable comparison also needs a saved case set and an identified release that another person can run.

Attach capacity evidence after quality

Once the candidate's task behavior is acceptable, measure it under representative lengths, request arrival and concurrency. Record the accepted quality policy with those tests. A faster quantized release that fails the task should not win simply because its tokens-per-second number is larger.

Quality and speed can interact. A shorter answer may save generation work while omitting necessary evidence. A longer retrieved prompt may improve support and increase prompt processing. Measure the full task policy, then explain which configuration changed and how the results differ.

If another model grades the answers, check its judgments against trusted labels on a sample and record where they disagree. Save the judge version with the run so a later change in grading does not look like a change in answer quality. Sensitive outcomes may also need human review. The evaluation should give the person approving the release enough evidence to explain that decision.

Do this before moving on

Create an evaluation sheet with the 100-request funnel and the paired table. Calculate 90%, 72% and 78%, then write two reasons you might reject the candidate despite its higher overall pass rate. Add a permission-sensitive slice and one unsupported question to the case list. State who decides whether the cited passage actually supports the answer. Add the east/west fixture and record its passing format checks and failed task check separately.

Handle the interview follow-up

If asked whether the faster model is better, state the acceptance rule before comparing rates. In this example, 90% quality among completed requests hides the fact that only 72 of 100 offered requests produced a qualifying result. Ask to compare both candidates on the same cases, including failures and deadlines. A higher completed-answer score would change your choice only if the service also meets its required useful-outcome target.

Go deeper

• Evaluation Metrics and Data Splits derives the denominators and threshold trade-offs used here.
• Serving Benchmarks That Do Not Lie turns a quality-qualified candidate into a reproducible load experiment.
• Evaluation and Data Pipeline Infrastructure preserves case identity and results across larger runs.
• Structured Outputs and Constrained Decoding traces token constraints and request state in the Premium reference.

Key takeaways

• Ninety percent quality among completions can coexist with 72% useful on-time output.
• Paired results expose ten gains and four regressions behind a six-point increase.
• Valid JSON and a passing schema can still request the wrong region.
• Keep evaluation identity attached to every release comparison.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 181 of 90 completions pass quality; 72 of 100 offered requests also meet the deadline. What is useful on-time output?

   72%90%81%

2. 2Why inspect four regressions when the candidate gains ten other cases?

   Answer it, then reveal

3. 3The user asks for east. The result says west, and the schema permits either. What failed?

   JSON parsingThe schema, because every wrong answer violates itTask correctness; the schema permits a value the user did not request

4. 4A streamed body already parses as JSON, but the request ended at its output limit. Is that enough to invoke the tool?

   Answer it, then reveal