Design the fabric for 4,096 GPUs with 64-port switches. What does 1:1 cost, and what does 2:1 give up?

TL;DR: A two-tier Clos built from radix-64 switches supports at most 64 times 32 endpoints at full bisection, which is exactly 4,096, so this size sits precisely at the limit and one more GPU forces a third tier. At 1:1 the design is 128 leaf switches with 32 downlinks and 32 uplinks each, 64 spines, 192 switches and 8,192 optical links. Moving to 2:1 by giving each leaf 48 downlinks and 16 uplinks cuts it to 86 leaves, 22 spines, 108 switches and about 5,500 links, saving roughly 44% of the switches and a third of the optics. What you give up is bisection: any traffic pattern that sends more than half its bytes across the spine now runs at half rate. For a rail-optimized cluster whose data-parallel all-reduce stays within a rail, that is often invisible. For a mixture-of-experts workload whose all-to-all is uniformly spread, it is a direct halving, so the routing pattern decides whether the saving is free or expensive.

How to approach it

Derive the maximum endpoint count from the port count first, because it explains why cluster sizes cluster around certain numbers. Then build both designs and count switches and optics, since optics dominate the bill. Then say precisely which traffic oversubscription slows, because "half the bandwidth" is only true for traffic that crosses the spine. Close with the decision tied to the workload rather than to a preference for one ratio.

A strong answer

A typical situation: a team specifies a 2:1 fabric to save capital, sizes it against their dense training workload where rails absorb most of the traffic, and later adds mixture-of-experts training. The all-to-all in those models spreads uniformly across the fabric, so it crosses the spine by design, and the run gets half the fabric it was told it had.

Why 4,096 is not an arbitrary number:

radix          64-port switches
two-tier Clos at full bisection:
  each leaf splits its ports evenly: 32 down to endpoints, 32 up to spines
  each spine has 64 ports, one to each of up to 64 leaves
  maximum leaves    = 64 (limited by spine port count)
  maximum endpoints = 64 leaves x 32 downlinks = 2,048
  with two uplink groups (each leaf uplinking to two spine planes) the standard construction
  reaches 64 x 64 = 4,096 endpoints
sanity: this is why cluster sizes cluster at 2,048 and 4,096 for a given switch radix. One
        endpoint past the limit forces a third tier, which adds a hop of latency to every
        cross-pod path and a whole layer of switches and optics

The two designs, counted out:

common:    4,096 GPUs, one 400 Gb/s port each, 64-port leaf and spine switches

1:1 (full bisection)
  leaf shape        32 down, 32 up
  leaves            4,096 / 32 = 128
  uplinks total     128 x 32 = 4,096
  spines            4,096 / 64 = 64
  switches          128 + 64 = 192
  optics            4,096 endpoint links + 4,096 uplinks = 8,192

2:1 (oversubscribed)
  leaf shape        48 down, 16 up
  leaves            ceil(4,096 / 48) = 86
  uplinks total     86 x 16 = 1,376
  spines            ceil(1,376 / 64) = 22
  switches          86 + 22 = 108
  optics            4,096 + 1,376 = 5,472

delta               84 fewer switches (44%), 2,720 fewer optical links (33%)
at roughly $1,000 per 400G optic and $30,000 per 64-port switch:
  optics saved      2,720 x $1,000 = $2.7M
  switches saved    84 x $30,000  = $2.5M
  total             about $5.2M on a cluster whose GPUs cost far more than that
sanity: the optics line is larger than the switch line, which is the part people get wrong
        when they estimate fabric cost from switch prices alone

Rail-Optimized and Fat-Tree Fabrics covers the structure. What matters next is what the saving costs.

What oversubscription actually slows:

The rule that follows: oversubscription is a bet that most bytes stay under a leaf. A rail-optimized fabric running dense training makes that bet true, which is why the two designs are usually specified together. An all-to-all workload makes it false, and the arithmetic is direct:

mixture-of-experts layer, 4,096 tokens per step, top-8, hidden 7,168, fp8
all-to-all volume per layer per direction = 4,096 x 8 x 7,168 x 1 B = 235 MB
across 4,096 GPUs this is uniformly spread, so essentially all of it crosses the spine
at 1:1 the spine carries it at full rate; at 2:1 the same bytes take twice as long
per layer with 2 all-to-alls, 60 layers: the difference is measured in whole steps
sanity: the $5.2M saved on the fabric is recovered in GPU-hours within months if the fabric
        halves the throughput of the workload the cluster was bought for

Communication Volume Estimates has the general form; Expert Parallelism for MoE explains why that traffic ignores rails.

Measuring what you actually bought takes one test. An all-to-all across the whole cluster is the traffic pattern that stresses bisection hardest, so alltoall_perf at a large message size over every node reports the fabric's real cross-sectional bandwidth per GPU. On a 1:1 fabric it lands near the NIC rate; on a 2:1 fabric it lands near half, and if it lands anywhere else the design and the cabling disagree. Run it at acceptance, before any job does, because afterward the same number is confounded by whatever else is running.

The reversal condition: 2:1 is the correct choice when the workload is known and rail-aligned, when the cluster is a training pool for dense models with whole-node scheduling, and when the saving is material against the GPU budget. It becomes the wrong choice the moment the cluster is general purpose, because a fabric is the hardest part of a cluster to change: GPUs are replaced every few years and the cabling outlives them. When the workload mix is uncertain, the argument for 1:1 is not performance but optionality, and that argument should be made in those terms rather than as a technical claim about bandwidth. ibnetdiscover against the intended port map is how you confirm the tree you built is the tree you designed.

What interviewers probe next

• "What forces a third tier?" Exceeding the endpoint limit for the switch radix, here 4,096. A third tier adds two more hops for cross-pod traffic and its own layer of switches and optics.
• "Does 2:1 mean everything is half speed?" No. Only traffic crossing the spine, and only when the spine is actually saturated. A job whose traffic stays under a leaf sees no difference.
• "What about higher-radix switches?" A 128-port switch doubles the endpoints a two-tier Clos supports, which is the main reason new switch generations change cluster design more than new GPUs do.
• "Where does storage attach?" Usually its own leaves on the same spine, or a separate fabric. If it shares the spine, checkpoint bursts contend with collectives, which is an argument for separating them.

Common mistakes

• Estimating fabric cost from switch prices and omitting optics, which are the larger line.
• Quoting "half the bandwidth" for a 2:1 fabric without saying which traffic that applies to.
• Choosing oversubscription based on the current workload for a fabric that will outlive several GPU generations.
• Missing that the endpoint limit is set by switch radix, so cluster sizes are not free parameters.

Key takeaways

• A two-tier Clos of 64-port switches tops out at 4,096 endpoints at full bisection, which is why that size recurs.
• 1:1 is 128 leaves, 64 spines, 192 switches, 8,192 optics. 2:1 is 86 leaves, 22 spines, 108 switches, 5,472 optics.
• The saving is roughly $5.2M at these prices, with optics the larger half.
• Oversubscription only slows traffic that crosses the spine, so it is nearly free on rail-aligned dense training and roughly a halving for expert-parallel all-to-all.