operations

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

The spares number falls out of the failure rate times the repair turnaround, and the cheap items are the ones people forget. The arithmetic for GPUs, the very different arithmetic for transceivers, and the rule that decides whether to hold a part at all.

The rule is one sentence and the execution is thousands of connections where a single transposition creates a fault that is invisible until a collective runs. The mapping, the labelling scheme that makes errors findable, and the verification that has to happen before anyone accepts the cluster.

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

A check that is too strict removes healthy capacity and one that is too loose lets a broken node take a job down. The three tiers by cost, the fleet-median comparison that catches degradation no threshold does, and the false-positive budget that has to be set before any of it runs.

The hardware is identical and almost everything around it moves: who can touch a node, how long a swap takes, what the contract actually guarantees, and what you can no longer measure. The six things to negotiate before signing and the two that decide whether operations work.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Both run at the same port speed, so the discount is not buying slower hardware. It is buying a different operational model and a different failure mode, and whether that is worth 15 percent depends on what your team already runs and what the fabric has to carry.

Measure the hours before you rank anything, because the task that feels worst is usually not the one that costs most. The ordering rule that keeps automation from causing the outage it was meant to prevent, and why detection always ships before remediation.