quality

Fluent and wrong is a plumbing failure, not a model failure, because a model whose weights are broken produces obvious nonsense. Five places the plumbing goes wrong, the reproduction that isolates each, and the one that is invisible until an evaluation runs.

A model update is a deploy whose failures are invisible to every deployment metric. What has to be re-validated even for a point release, the canary that watches output shape rather than error rate, and the fingerprint that makes a customer report investigable.

The cheapest correct answer beats the best answer at scale, so routing is a cost problem with a quality floor. The three routing signals that work, the cascade that trades latency for cost, and the measurement that catches quality loss no latency dashboard shows.

The conversion is the easy part and the evaluation is the project. What calibration data does and why yours should come from production, the per-category evaluation that catches what an average hides, and the parts of the model that should not be quantized at all.

Model evaluations answer whether the weights are good and deployment evaluations answer whether your configuration serves them correctly, which is a different and more common failure. What to sample, why category averages hide the regressions that matter, and the reference that makes a result mean something.