Measure useful answers before renting more compute

TL;DR: Evaluate the task outcome before accepting a speed improvement. Preserve failures and late responses in the denominator, compare releases on the same cases, and keep threshold selection separate from the final test.

Where you are. You can identify a deployed release. Now decide what evidence would make you approve it. You do not need to train a model to perform this exercise.

A successful response can contain an unusable answer

The documentation service returns text for a question. That proves it produced output; it does not prove the output answers the question or cites permitted, current evidence. Write an acceptance rule before collecting a benchmark.

For our exercise, an acceptable answer must address the question, be supported by its cited passage and arrive before the stated deadline. Unsupported questions should receive an appropriate abstention rather than invented evidence. The exact rubric depends on the task. Keep model quality, authorization and timing as separate recorded checks even when a final acceptance condition combines them.

These hypothetical counts conserve the original population: 72 acceptable-on-time, 9 acceptable-late, 9 wrong and 10 unfinished make 100. Completed-answer quality is 81/90 = 90%. Acceptable-on-time service output is 72/100 = 72%. Both are useful, but substituting one for the other hides the cost of failures and delay.

Compare the same cases

Keep a stable case identifier, input, reference evidence, permission scope and expected evaluation behavior. Run baseline and candidate on the same cases under the same stated policy. Save outputs and failures with release identities. If a request is retried, retain attempt records but count the logical request once for the user-facing result.

Suppose 100 paired cases produce this table, where success means passing the full acceptance rule:

Baseline success is 72%; candidate success is 78%. The candidate gained ten cases and lost four. Inspect those four regressions before deciding the six-point aggregate increase is worthwhile. A failure on a permission-sensitive question can matter more than several minor formatting improvements. This small example does not establish statistical significance.

Learn the metric that matches the decision

For a binary routing model, precision and recall help explain false alarms and missed cases. For retrieval, measure whether the needed source was retrieved. For generated answers, check support and task correctness using a suitable rubric. A text-similarity score is not automatically a correctness measure.

A threshold determines which score values trigger an action. Select it using validation data and the relevant error costs. Keep the final test set separate from this choice. Group related documents or conversations across the split so near-duplicate content does not turn the test into a memory exercise.

The minimum ML workflow is input representation, model computation, output interpretation and evaluation. A deep-learning framework supplies operations for training and inference; it does not decide whether your labels, task definition or metric are appropriate. A notebook can help inspect examples, but the repeatable result needs a pinned release and case set outside its current interactive state.

Attach capacity evidence after quality

Once the candidate's task behavior is acceptable, measure it under representative lengths, request arrival and concurrency. Record the accepted quality policy with those tests. A faster quantized release that fails the task should not win simply because its tokens-per-second number is larger.

Quality and speed can interact. A shorter answer may save generation work while omitting necessary evidence. A longer retrieved prompt may improve support and increase prompt processing. Measure the full task policy, then explain which configuration changed and how the results differ.

Do not treat an automated judge as ground truth without examining its errors. Compare it with trusted labels on a sample, record disagreement, and keep the judge version with the run. For sensitive outcomes, human review may be part of the release process. The goal is an accountable decision, not the largest possible score table.

Do this before moving on

Create an evaluation sheet with the 100-request funnel and the paired table. Calculate 90%, 72% and 78%, then write two reasons you might reject the candidate despite its higher overall pass rate. Add a permission-sensitive slice and one unsupported question to the case list. State who decides whether the cited passage actually supports the answer.

Handle the interview follow-up

If asked whether the faster model is better, state the acceptance rule before comparing rates. In this example, 90% quality among completed requests hides the fact that only 72 of 100 offered requests produced a qualifying result. Ask to compare both candidates on the same cases, including failures and deadlines. A higher completed-answer score would change your choice only if the service also meets its required useful-outcome target.

Go deeper

• Evaluation Metrics and Data Splits derives the denominators and threshold trade-offs used here.
• Serving benchmarks turns a quality-qualified candidate into a reproducible load experiment.
• Evaluation and Data Pipeline Infrastructure preserves case identity and results across larger runs.

Key takeaways

• Ninety percent quality among completions can coexist with 72% useful on-time output.
• Paired results expose ten gains and four regressions behind a six-point increase.
• A framework or notebook cannot choose a meaningful acceptance rule for you.
• Keep evaluation identity attached to every release comparison.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 181 of 90 completions pass quality; 72 of 100 offered requests also meet the deadline. What is useful on-time output?

   72%90%81%

2. 2Why inspect four regressions when the candidate gains ten other cases?

   Answer it, then reveal