deployment

Start from the power envelope, because it decides the GPU count more often than the budget does. The scalable unit that makes the design a copy of a known-good one, the four networks, and the three line items that gate the schedule while everyone is discussing GPUs.

Four separate facility limits have to be cleared and total building capacity is not one of them. Power per rack, heat rejection, floor loading and network reach each fail independently, and each has a different remedy with a different lead time. The options that exist, priced, and the one that is usually taken.

Links that never come up and links that come up and flap are different faults with different causes. The reach change between 400G and 800G explains most first-time failures, and the counters separate the rest. The order to check, and the one physical measurement that settles it.

The GPU board power is about 60 percent of the answer. The rest is CPUs, memory, NICs, fans, switches, thousands of transceivers and the cooling plant, and each is a term people leave out. The full derivation, the term that surprises everyone, and the contracted number that is larger than all of it.

Every watt in comes out as heat, so the design starts from the load and one equation. The flow rate, the CDU count and redundancy, the water quality requirements that decide whether the loop survives three years, and the automatic response that has to fire before a human can read the page.

Each layer depends on the ones below it and a mistake at the bottom shows up six layers higher as something else entirely. The order, the dependency almost everyone gets wrong on Blackwell systems, and the validation ladder that keeps a fault attributable to one node instead of five hundred.

Acceptance is a contract document, not a checklist, so every criterion has to be a number with a tolerance that a vendor can be held to. Comparing nodes against each other rather than against a datasheet is what finds real defects, and the duration is what finds the thermal ones.

A training cluster is a synchronized load, so the peak is far above the average and the average is what everyone budgeted. The immediate mitigation takes minutes and costs a few percent of throughput, and the four durable fixes trade cost against throughput in different places.

The count follows from three inputs and one topology rule, and getting it wrong by the leaf-to-spine half is the classic error. The arithmetic, the radix check that says two tiers is enough, and the power the transceivers add that nobody put in the budget.

Single-node fine and two-node halved is a specific signature, and it points at one thing before anything else. Why the copy path costs exactly about half, the five causes that produce it, and the one log line that confirms which.

Capacity is the easy half and almost never the constraint. The number that sizes the system is the checkpoint write burst, which is thousands of times the steady dataset read, and the tier that serves each is different. The full derivation, and the tiering that avoids paying burst prices for archive bytes.

The endpoint count is larger than the node count and that surprises everyone building their first cluster. What has to be reachable, why this network must survive when every other one is down, and the security posture it needs because it can power-cycle the entire fleet.

Three approaches with overlapping ranges, and the choice is decided by the facility and the service model rather than by thermodynamics alone. Where each runs out, what the middle option buys that people underrate, and the operational cost that comes with the most capable one.

The GPUs are almost never the critical path and planning backward from their delivery date is how deployments slip. The real ordering, what can run in parallel, the two dependencies that cannot, and the cost of a month of delay expressed in the unit that gets attention.

The spares number falls out of the failure rate times the repair turnaround, and the cheap items are the ones people forget. The arithmetic for GPUs, the very different arithmetic for transceivers, and the rule that decides whether to hold a part at all.

A 44 percent failure rate is far above any plausible hardware defect rate, so the first hypothesis is that the test or the environment is wrong rather than the hardware. The three checks that separate those, the cohort analysis that localizes a real defect, and how to run the vendor conversation.

Most architecture diagrams show one network and a cluster has four. Each has a different traffic pattern, a different failure consequence and a different port count, and the one that is under-counted every time is the one you need when everything else is broken.

Switch radix decides it, and at 10,000 endpoints the answer sits right at the boundary for current high-radix switches. The arithmetic, what a third tier costs in switches, optics and power, and the two designs that avoid it entirely.

Same GPU count, same silicon, and a completely different machine. The domain size is the whole argument, it is worth an order of magnitude on specific traffic, and it costs a failure domain nine times larger plus a facility that most halls do not have.

The break-even is a utilization number, not a price comparison, and most teams overestimate the utilization they will actually reach. The full cost of owning, the number that decides it, and the two situations where renting wins even at high utilization.

A fixed power envelope turns hardware selection into an optimization with one objective and several overheads competing for the same watts. Where the 1 MW actually goes, why the answer is not simply the most efficient GPU, and the two overheads that can be cut to buy more compute.

It configures the NVSwitch fabric so eight GPUs present as one NVLink domain, and when it is absent nothing errors. Jobs run, results are correct, and everything multi-GPU is slower by a factor the team spends weeks not finding. The check that takes one command.

Isolation has to be physical where it matters and logical where it can be, and picking the boundary wrong is either expensive or a security problem. What partitions cleanly, what does not, and the allocation unit that decides fragmentation and margin.

The rule is one sentence and the execution is thousands of connections where a single transposition creates a fault that is invisible until a collective runs. The mapping, the labelling scheme that makes errors findable, and the verification that has to happen before anyone accepts the cluster.

Acceptance measures the hardware and production measures the whole system, so the gap lives in everything acceptance did not cover. Four candidate layers, the order to eliminate them, and the two that account for most of these cases.

A fleet-wide version change is the most likely cause of the next unexplained performance regression, so the rollout is designed to make that attributable. Cohorts, a canary that measures rather than boots, and the rollback that has to be real before the first node is touched.

Almost nothing about a generation change is the GPU. The facility usually cannot take the new part in the old positions, the fabric generation may not match, and the two fleets have to coexist for months. The sequencing that avoids a capacity trough, and the decision people get wrong.

Peak numbers are almost never the reason a part wins or loses, and the software stack decides more deployments than the silicon does. The four things to measure, the question that eliminates most challengers in one conversation, and the pilot that gives a defensible answer.

A check that is too strict removes healthy capacity and one that is too loose lets a broken node take a job down. The three tiers by cost, the fleet-median comparison that catches degradation no threshold does, and the false-positive budget that has to be set before any of it runs.

Two failures and a pass is not a passing node, it is a node with a demonstrated intermittent fault and a small sample. The arithmetic that shows how weak the evidence for acceptance is, what an intermittent fault costs once it is in a training job, and the policy that removes the argument.

Doubling per-rack density is four separate upgrades with different lead times, and the electrical one usually cannot be done while the hall runs. The phasing that keeps capacity available through it, the constraint that decides whether it is possible at all, and the number that says whether to upgrade or move.

The hardware is identical and almost everything around it moves: who can touch a node, how long a swap takes, what the contract actually guarantees, and what you can no longer measure. The six things to negotiate before signing and the two that decide whether operations work.

Modules do not fail randomly at three times a published rate, so the excess has a cause and it is usually thermal or a batch. The cohort cuts that localize it, the counters that separate a dying module from a dirty connector, and the environmental cause that looks like a hardware defect.

Firmware skew produces intermittent faults that surface layers away from their cause, so an inventory is a debugging tool rather than a compliance exercise. What to collect, how drift appears without anyone deploying anything, and the diff that turns a week-long investigation into a query.

Both run at the same port speed, so the discount is not buying slower hardware. It is buying a different operational model and a different failure mode, and whether that is worth 15 percent depends on what your team already runs and what the fabric has to carry.

Older parts are rarely worthless and often better suited to work the new fleet should not be doing. The four options ranked by what they return, the data handling that has to happen before any hardware leaves, and the reason the answer is usually not one option.

A whole rack failing at once is a shared dependency, which narrows the causes to four before anyone touches anything. What the out-of-band network tells you in the first minute, why the order of restoration matters, and the decision about liquid cooling that has to be made before power returns.

Model evaluations answer whether the weights are good and deployment evaluations answer whether your configuration serves them correctly, which is a different and more common failure. What to sample, why category averages hide the regressions that matter, and the reference that makes a result mean something.

A 141 GB artifact is not a Docker image, and shipping it like one costs an hour per node and a registry that melts under a rollout. Content-addressed chunks, a manifest per version, a fan-out that turns one origin into a thousand nodes in minutes, integrity that fails closed, and promotion tied to evaluation.