fabric

Start from the power envelope, because it decides the GPU count more often than the budget does. The scalable unit that makes the design a copy of a known-good one, the four networks, and the three line items that gate the schedule while everyone is discussing GPUs.

A rail-optimized fabric makes this fault findable because the eight rails are identical by construction, so any asymmetry is a defect. Bisecting by rail, then by switch, then by node, then by port narrows 512 links to one in four steps, and the counters say which of three physical causes it is.

The count follows from three inputs and one topology rule, and getting it wrong by the leaf-to-spine half is the classic error. The arithmetic, the radix check that says two tiers is enough, and the power the transceivers add that nobody put in the budget.

Most architecture diagrams show one network and a cluster has four. Each has a different traffic pattern, a different failure consequence and a different port count, and the one that is under-counted every time is the one you need when everything else is broken.

Switch radix decides it, and at 10,000 endpoints the answer sits right at the boundary for current high-radix switches. The arithmetic, what a third tier costs in switches, optics and power, and the two designs that avoid it entirely.

A fixed power envelope turns hardware selection into an optimization with one objective and several overheads competing for the same watts. Where the 1 MW actually goes, why the answer is not simply the most efficient GPU, and the two overheads that can be cut to buy more compute.

Isolation has to be physical where it matters and logical where it can be, and picking the boundary wrong is either expensive or a security problem. What partitions cleanly, what does not, and the allocation unit that decides fragmentation and margin.

The rule is one sentence and the execution is thousands of connections where a single transposition creates a fault that is invisible until a collective runs. The mapping, the labelling scheme that makes errors findable, and the verification that has to happen before anyone accepts the cluster.

Modules do not fail randomly at three times a published rate, so the excess has a cause and it is usually thermal or a batch. The cohort cuts that localize it, the counters that separate a dying module from a dirty connector, and the environmental cause that looks like a hardware defect.

Both run at the same port speed, so the discount is not buying slower hardware. It is buying a different operational model and a different failure mode, and whether that is worth 15 percent depends on what your team already runs and what the fabric has to carry.

Three networks, not one, and the mistake that hurts most is putting two of them on the same wires. The port and switch counts for each, the cable plan, the split of the bill between switches and optics, and the two decisions that are hard to change once the cluster is racked.

It depends entirely on which dimension you split the model along, and the three choices differ by two orders of magnitude in what they demand. The per-token traffic for each, the arithmetic that puts tensor parallelism on the fast link and pipeline parallelism anywhere, and where disaggregated serving lands.

A synchronous training job across two clouds runs at the speed of the link between them, a thousand times slower than the fabric inside either. The arithmetic that kills the idea for one job, the two cases where it is tolerable, and what a platform does instead: partition the work, not the job, and move data ahead.

Ten thousand GPUs is 1,280 nodes, ten pods, about 700 switches, a filesystem that swallows a 5 TB checkpoint in under a minute and a 14 MW feed. The bill of materials with a reason on every line, from the FLOPs the model needs to the spare pool the failure rate demands.