Your new model deployment produces fluent answers that score badly on evaluations. Where do you look?

TL;DR: Fluent text can be wrong because of the model, numerics, request construction, generation settings, parsing or scoring. Freeze the failing examples and both deployment manifests. Compare the input token IDs first, then effective generation settings, raw generation, parsed response and evaluator result. A model-card example is a useful fixture; matching it does not certify the deployment, and different sampled text does not prove a bug.

How to approach it

Start with what changed and a reproducible failing request. State the reference: a previously validated deployment or the authors' supported implementation at the same checkpoint revision. Explain which comparison would narrow the fault at each boundary. Finish with a regression suite and a rollout decision tied to the affected task class.

A strong answer

A new deployment answers ordinary questions well, but tool tasks lose accuracy and long responses sometimes contain trailing text. First preserve the requests, expected outcomes, returned objects and finish reasons (why generation stopped: an end token, the length limit or a stop string). Compare the evaluated population too: a changed test set, answer extractor or timeout policy can lower the score without any model change.

Pin the model, tokenizer, template, engine, parser and evaluation revisions. Record resolved sampling and stopping settings rather than just the flags someone remembers passing. Run the old and new paths on the same fixtures. Avoid storing private prompts in unrestricted debug logs.

Find where reference and candidate first differ

Keep the same failing request and pinned deployment versions through each comparison.

1. Input and settings

   Compare rendered token IDs, tool schemas, effective sampling and stop rules.

2. Raw generation and parsed response

   Check what the model emitted before judging what the API parser returned.

3. Evaluator decision

   Compare the parsed result with the scoring rule, preserving failures and timeouts.

Make the input comparison exact

Use the model's supported conversation encoder. Compare token IDs and an escaped rendering of the prompt: whitespace is visible in a careful string diff, while token IDs expose special-token and segmentation differences directly. Check the system turn, assistant-generation prefix, tool definitions and every branch used by the failing request. A text-only fixture does not test image preprocessing or a tool-result turn.

Hugging Face's chat-template guide warns against adding special tokens again after rendering a template that already supplies them. With its API, tokenizing inside apply_chat_template avoids that particular double-tokenization mistake. Starting a new assistant message and continuing a partially filled message are different operations; their options cannot simply be combined.

Inspect effective stopping behavior

A token does not become an ordinary token because a stop rule omitted it. It can remain a special token while generation continues past it. Conversely, the engine may already import multiple end IDs from the model's generation configuration. In vLLM 0.28, generation_config="auto" loads the model's configuration; choosing "vllm" changes that source of defaults.

Inspect the resolved configuration, ignore_eos, minimum-token settings, request stop strings and output limit. Exercise the intended end-of-turn and tool-call paths separately. Do not make every special token a stopping condition: some delimiters belong inside a structured response. Check whether termination was an end token, a configured stop, a length limit or cancellation.

Suppose a controlled synthetic reproduction shows that fixing a stopping defect reduces the mean generated length from 640 to 220 tokens without removing useful output:

extra tokens per request = 640 - 220 = 420
at a hypothetical 40 completed requests/s:
  useful output = 40 × 220 = 8,800 tokens/s
  total output  = 40 × 640 = 25,600 tokens/s
  extra output  = 40 × 420 = 16,800 tokens/s
length multiplier = 640 / 220 = 2.91
extra share       = 420 / 640 = 65.6%
sanity: 8,800 + 16,800 = 25,600; both ratios use the same population.

That is a token-work illustration, not a measured fleet cost multiplier. Decode cost varies with context, batching and cache behavior, and a fleet might fail to sustain the assumed completion rate. Extra generation can also increase queueing and latency; a formatting defect is not necessarily invisible on capacity dashboards.

Separate runtime differences from task limitations

If inputs and settings match but quality differs before parsing, compare the same checkpoint with a validated runtime and precision path. Change one variable at a time. Requantization is a strong suspect when it is the deployment change, but the authors' released format is not a guarantee of quality on your tasks. Numeric defects and normal model limitations can both produce fluent text. Neither must affect every prompt, and there is no universal order in which code, long context or short answers degrade.

At an identical prefix, compare logits with a tolerance suited to the implementation rather than demanding bitwise equality across different kernels. Greedy decoding reduces sampling variation but small numeric differences near a tie can still change a token and all subsequent text. Use task outcomes over repeated or sufficiently broad fixtures to judge material quality loss. A single exact-output comparison cannot establish that loss.

Default to holding or rolling back a candidate that fails an important contract. If replay shows that the candidate meets the contract and the evaluator made a mistaken assumption, fix the test harness and rerun both versions before changing the deployment. Model Onboarding: From Hugging Face to Production covers the release gates; Serving Benchmarks That Do Not Lie covers measuring the resulting service.

What interviewers probe next

• "The card example matches. Are we done?" No. It covers one path; add multi-turn, tool, streaming, stopping and relevant context-length fixtures, then the actual task evaluation.
• "The raw tool call is correct but the API has no tool call. What next?" Compare parser selection and streaming reconstruction, including delimiters split across chunks. Validate argument values and schema, not just JSON syntax.
• "Can marker detection catch this?" It can flag unexpected delimiters, but legitimate quoted examples can contain them. Condition the signal on request type and confirm with raw output and finish reasons.
• "Why not compare to the published score directly?" Match the checkpoint, data split, prompt format, decoding budget, tools and scoring procedure first. Otherwise several differences are mixed together.

Common mistakes

• Treating fluency as evidence for or against a weight/runtime defect.
• Certifying the entire deployment from one matching sample.
• Assuming the model configuration and effective request settings are identical.
• Treating a missing stop rule as a change to the tokenizer's special-token vocabulary.
• Changing precision, template and evaluator simultaneously, then attributing the result to one change.

Key takeaways

• Compare five boundaries: model input, settings, generation, parsing and scoring.
• A 220-to-640-token expansion adds 420 tokens per request; 65.6% of the resulting output is extra in the stated example.
• Branch-specific fixtures catch defects that a plain chat example cannot.
• Use controlled references and task-level evidence; exact sampled text is not a general correctness oracle.