Multi-Node Serving Topologies
Once a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.
TL;DR: Keep tensor parallelism inside the NVLink domain and choose what crosses the fabric. Replication is the default and the simplest: independent copies of the model behind a router, scaling linearly and failing independently, viable whenever the model fits in one domain. Pipeline parallelism splits layers across nodes and communicates one activation per micro-batch boundary rather than per token, so it tolerates the fabric, at the cost of a bubble that grows with stage count. Expert parallelism spreads experts across nodes, which is efficient when the all-to-all backend matches the interconnect and poor when it does not. Disaggregation runs prefill and decode on separate pools sized independently and moves the KV cache between them over the fabric, which is the shape that lets each phase be tuned for its own bottleneck. In vLLM the multi-node machinery is
--data-parallel-sizewith--data-parallel-size-local,--data-parallel-start-rank,--data-parallel-addressand--headlesson the secondary nodes, plus--kv-transfer-configwhen disaggregating.
The four shapes
| Shape | What crosses the fabric | Scales with | Main failure mode |
|---|---|---|---|
| Replication | Nothing but requests | Linear in replicas | None interesting; a replica dies and the router routes around it |
| Pipeline parallel | One activation per micro-batch boundary | Sub-linear; the bubble grows with stages | One slow stage stalls the whole pipeline |
| Expert parallel | Two all-to-alls per MoE layer per token | Well inside a domain, poorly across a slow fabric | Expert load imbalance sets the step time |
| Disaggregated | The KV cache of each finished prefill | Each pool independently | KV transfer bandwidth becomes the coupling |
When the model does not fit in one domain
decision arithmetic, using GLM-5.3 in FP8 at about 760 GB of weights
on 8 x B300 (2,304 GB in one NVLink domain)
weights 760 GB, so 1,544 GB left for KV, activations and overhead
it fits in one domain -> REPLICATE, and never cross a node boundary for this model
on 8 x H100 (640 GB in one NVLink domain)
weights 760 GB do not fit at all
options
pipeline across 2 nodes: layers 0-38 on node 1, 39-77 on node 2
one activation crosses per micro-batch, hidden 6,144 x 2 B
= 12 KB per token, trivial for the fabric
cost: the bubble, and every token traverses both nodes
tensor parallel across 2 nodes: 4.6 MB per token per GPU of all-reduce crossing the
fabric at 100 GB/s per GPU instead of 900 GB/s of NVLink
this is the choice to avoid
sanity: pipeline moves 12 KB per token across the fabric and cross-node tensor parallelism
moves 4.6 MB, a factor of 380, which is the whole reason the layer split beats the
width split once you leave the domain
Pipeline Parallelism and the Bubble covers the cost side. For serving specifically, the bubble matters less than it does in training because requests arrive continuously and the scheduler can keep stages fed, which is why pipeline parallelism is more attractive for inference than its reputation from training suggests.
Wiring a multi-node deployment in vLLM
one primary node and N secondaries, data-parallel across nodes
primary:
--data-parallel-size <total DP across all nodes>
--data-parallel-size-local <DP processes on this node>
--data-parallel-address <primary IP>
--data-parallel-rpc-port <a reachable port>
--api-server-count <scale to the local rank count>
each secondary:
--headless worker only, no API server
--data-parallel-start-rank <cumulative local DP of the nodes before it>
--data-parallel-address <primary IP> same address and port as the primary
--data-parallel-size-local <DP on this node>
and for a mixture-of-experts model
--enable-expert-parallel EP_SIZE = TP_SIZE x DP_SIZE
--all2all-backend <matched to the interconnect>
sanity: --data-parallel-start-rank is the field that is wrong most often, because it is a
cumulative count rather than a node index, and getting it wrong produces a cluster
that forms and then hangs on the first collective
Disaggregation, and when it is worth the complexity
why the phases want different machines
prefill: compute-bound, benefits from large batches and high FLOPS, tolerates latency
decode: bandwidth-bound, benefits from many concurrent sequences and high memory
bandwidth, and is latency-critical
running both in one pool means every configuration is a compromise between them
what disaggregation costs
the KV cache of a finished prefill has to reach the decode pool
for GLM-5.3 at 87.75 KB per token, a 4,096-token prompt produces
4,096 x 89,856 = 368 MB of KV to transfer
over a 100 GB/s per-GPU fabric link that is 3.7 ms, which is small against a prefill that
took hundreds of milliseconds
compare a model with classic attention at 320 KB per token:
4,096 x 327,680 = 1.34 GB, or 13.4 ms
sanity: compressed-latent attention makes disaggregation roughly four times cheaper to move,
which is a reason these architectures and this serving shape appeared together
In vLLM the connector is configured through --kv-transfer-config, naming the connector and
whether the instance is a producer, a consumer or both. Disaggregated Prefill and Decode
covers the design; the topology point is that disaggregation is the only shape that lets the
two pools be sized and tuned independently, and that its coupling cost is a function of the
model's KV per token.
What interviewers are listening for
The rule that tensor parallelism stays in the domain, stated with the number that justifies it. After that, whether the candidate picks the shape from the constraint rather than from fashion: replicate when it fits, pipeline when it does not, expert-parallel when the model is sparse and the interconnect allows, disaggregate when the two phases want different machines and the traffic justifies the extra moving parts and the traffic is large enough to pay for the extra moving parts. Interviewers also like the KV transfer arithmetic, because it converts "disaggregation is expensive" into a millisecond number that can be compared against the prefill it replaced.
Key takeaways
- Tensor parallelism stays inside the NVLink domain; crossing it moves 4.6 MB per token where a pipeline split moves 12 KB, a factor of 380.
- Replicate when the model fits in one domain, which is the simplest shape and the one with no interesting failure mode.
- Multi-node vLLM uses
--data-parallel-size,--data-parallel-size-local,--data-parallel-start-rankand--headless, and the start rank is cumulative rather than a node index. - Disaggregation moves 368 MB of KV for a 4,096-token GLM-5.3 prefill, about 3.7 ms, against 1.34 GB for a classic-attention model of similar depth.
- Pipeline parallelism suits serving better than its training reputation suggests, because continuous request arrival keeps the stages fed.