quality

The answers sound fluent, but the scores dropped. Compare inputs, generation settings, raw output, parsing and scoring to find where the new deployment differs.

A minor revision can change weights, encoding or response behavior. Compare pinned deployment bundles, evaluate task slices, monitor a representative canary and keep enough capacity to restore the old service.

Separate pre-routing from a sequential cascade, charge every attempted generation, and measure checker mistakes. A lower escalation rate is not proof of higher quality, and the largest model is not a correctness oracle.

Define whether half the footprint means checkpoint bytes or runtime memory. Choose a supported recipe, measure which tensors it changes, separate calibration from held-out evaluation, and validate quality and serving cost.

Test the complete serving bundle with contract fixtures, representative tasks and load-aware outcomes. Preserve paired results, calibrate the scorer and distinguish a relative improvement from meeting an absolute product requirement.