canary

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

Two steps consume most of the week and neither is the one people plan for. The gate that can end the project on day one, the multi-hour download, the correctness check almost nobody runs, and the canary that has to see real traffic before Friday.

A model update is a deploy whose failures are invisible to every deployment metric. What has to be re-validated even for a point release, the canary that watches output shape rather than error rate, and the fingerprint that makes a customer report investigable.

A driver upgrade touches every layer at once: kernel module, CUDA runtime, container toolkit, NCCL, fabric driver, and every job's image. The compatibility matrix that decides whether a job can run on the new node, the canary that proves it, the wave arithmetic for 2,000 nodes, and the rehearsed rollback.

Roll back first and investigate afterward, because the error budget is being consumed while you read logs. The rollback decision rule that removes the argument, the four causes specific to model serving, and the canary design that would have caught it at one percent of the traffic.