fat tree

Switch radix decides it, and at 10,000 endpoints the answer sits right at the boundary for current high-radix switches. The arithmetic, what a third tier costs in switches, optics and power, and the two designs that avoid it entirely.

Wire GPU number three of every node to the same switch and the collective that matters most never leaves it. The rank mapping that makes the layout pay, the switch and cable count for a thousand GPUs worked out, and what happens to a job whose ranks are assigned in the wrong order.

Port counts decide everything: a two-tier Clos of 64-port switches tops out at exactly 4,096 endpoints at full bisection, which is why that number appears in so many cluster designs. The switch and optics count both ways, what oversubscription actually slows down, and where the money goes.