Multi-Node Serving Topologies

TL;DR: Start with a supported layout that fits each rank's state, then measure its latency and throughput on the target workload. Replication adds complete serving copies; TP splits tensor work, PP splits layers, EP splits experts, and phase disaggregation separates prefill from decode. Favor fast links for frequent communication, but do not turn a topology preference into a rule that cross-node TP can never win.

Choose the distributed object

A typical situation: a model fits only across several devices, and the team must decide whether the next node should enlarge one replica or create another. These choices spend memory and communication differently. A physical node boundary may not coincide with the fast interconnect domain; inspect the actual paths and collective performance.

These are composable. A replica can combine TP and PP, and a disaggregated pool can contain several such replicas. In MoE deployments, “data parallel” attention workers may share an expert-parallel group rather than form independently failing full-model copies. Expert Parallel and All-to-All Backends explains that distinction.

Fit is necessary, not a complete choice

Check weights, live cache/recurrent state, temporary allocations and graph/runtime buffers per rank. Aggregate device capacity can hide one overfull stage. Embeddings, output heads and uneven layer types also make a half-by-layer-count split an imperfect memory or timing balance.

The vLLM 0.28 parallelism guide gives TP within a node and PP across nodes as a common starting layout. It also documents other deployment choices and uneven layer splits. Treat the recipe as a candidate to test, not proof that fitting within one node makes a wider replica pointless.

TP may improve an individual request's latency by using more compute, while replication may improve aggregate throughput. Cross-node TP can be viable with a suitable fabric and workload. The number and shape of collectives depend on the architecture and implementation; “two all-reduces per layer per token” is not a universal model of every engine.

Pipeline traffic is small only relative to a stated workload

Suppose one PP boundary sends a single BF16 hidden-state tensor of width 6,144 for each processed position. This is a synthetic payload model; implementations may exchange additional tensors or shard the payload:

activation bytes = positions × hidden width × bytes/element
one position    = 1 × 6,144 × 2 = 12,288 B = 12 KiB
4,096 positions = 4,096 × 6,144 × 2 = 50,331,648 B = 48 MiB
sanity: a prefill chunk has many positions; its boundary payload is not 12 KiB total.

During ordinary autoregressive decode, each next-token forward pass still crosses the stage boundaries. PP does not communicate only once per request. The next token of one sequence also depends on completing the previous token's forward pass, so continuous arrivals do not guarantee a full pipeline at low concurrency.

For an ideal isolated forward wave with two balanced stages, four independent microbatches and five milliseconds per stage, ignoring communication:

wave duration = (microbatches + stages - 1) × stage time
              = (4 + 2 - 1) × 5 ms = 25 ms
busy time per stage = 4 × 5 = 20 ms
stage utilization in this wave = 20 / 25 = 80%
with only one microbatch: 5 / 10 = 50%
sanity: filling and draining cost slots even with perfectly balanced stages.

Pipeline Parallelism and the Bubble covers scheduling details. Unequal stages, transfer time and autoregressive dependencies can reduce the realized benefit; measure the actual scheduler rather than applying the finite-wave formula to all serving traffic.

Separate phases when the measured benefit pays for the coupling

Disaggregated Prefill and Decode explains moving required state between pools. Prefill contributes directly to TTFT and does not inherently tolerate queueing. Neither phase has one fixed bottleneck across all batch sizes and architectures. Splitting can reduce interference and allow different configurations, but the pools remain coupled by arrivals, state transfer and backpressure.

For a synthetic transfer of 1 GiB over an achieved 50 GB/s bottleneck, the serialization component is about 21.5 ms. Queueing, registration, layout conversion, staging and receiver readiness can add time. Conversely, overlap can hide some work. Use transferred bytes at that bottleneck, not whole-model cache bytes divided by one arbitrarily chosen GPU link.

The vLLM disaggregated-prefill guide labels the feature experimental and emphasizes phase/tail-latency control rather than an automatic throughput increase. Verify the selected connector's model, state layout and parallelism compatibility. A prefill-side prefix hit does not remove transfer unless the receiver already has compatible state or the connector otherwise avoids moving it.

Deployment and failure behavior follow the layout

vLLM Server Arguments That Matter is the configuration entry point. TP/PP executors, DP orchestration and KV connectors serve different roles; --headless and DP rank flags are not a universal multi-node recipe. Record rank placement, artifacts, interfaces and connector configuration, then test a real forward pass and recovery.

The reversal condition: keep or return to a simpler combined layout if the split misses the service objective or fails recovery requirements. Replication also needs failure planning: shared storage, routing, power or an expert group can defeat assumptions of independence. Request Routing and Load Balancing for LLMs covers balancing queue delay with cache locality.

What interviewers are listening for

Identify which bytes and dependencies cross which link, why the chosen layout fits, and what changes when a stage or node fails. Compare a measured alternative rather than quoting a universal communication ratio.

Key takeaways

• TP, PP, EP, replication and phase separation can be combined; their failure domains differ.
• The example PP boundary moves 12 KiB for one position but 48 MiB for a 4,096-position chunk.
• Two balanced stages with four independent microbatches reach 80% utilization only under the stated finite-wave assumptions.
• Transfer size, critical-path delay and surviving service capacity decide whether a split helps.