Expert Parallel and All-to-All Backends

TL;DR: Tensor parallelism shards expert matrices; expert parallelism places different experts on different ranks and routes token activations to them. Both can execute only the selected experts. EP does not create sparsity that TP lacks. The performance trade is matrix shape, local batch size, communication, state placement and imbalance. Match the supported backend to the actual topology and workload, then measure the critical path. Load balancing can rearrange or replicate experts, with a calculable memory and transfer cost.

Compare the same sparse computation

A mixture-of-experts (MoE) layer holds many expert feed-forward blocks, and a router sends each token to only a few of them. A typical situation: an MoE deployment moves from TP to EP expecting a reduction proportional to total experts divided by active experts. It gets slower. The expectation confused expert selection with expert placement.

Consider a synthetic layer with eight experts, two selected per token and four ranks. Let each expert contain 40 MB of weights, with no replicas or metadata in this example:

The selected payload row is an idealized count, not measured HBM traffic. Caching, packing and repeated reads can change actual traffic. EP can offer larger local matrices and avoid sharding each expert into small pieces, but it also needs enough token work across the group to keep ranks busy. TP can distribute a selected expert's computation across ranks. Expert Parallelism for MoE explains the placement axis in the training setting.

Size logical traffic before estimating link time

For a simple dispatch that sends a hidden vector to each selected expert, logical activation payload per layer and direction is tokens × top-k × hidden width × bytes. A return path of the same width and dtype doubles it. This counts expert assignments, including local ones; actual network traffic depends on co-location, deduplication, quantization, padding and routing metadata.

Synthetic example: 128 tokens, top-2, hidden width 4,096, BF16 dispatch and return:

One direction = 128 × 2 × 4,096 × 2 = 2,097,152 B = 2 MiB
Dispatch plus return = 4 MiB of logical payload per layer

The busiest link's bytes and the number of synchronization rounds matter more than this sum alone. Small decode messages can be latency-limited; large prefill messages can stress bandwidth and communication resources. Do not divide a whole-model payload by an advertised bidirectional NVLink figure and call it an observed all-to-all time. NVLink Domains and the NVL72 Rack explains why the domain boundary matters.

Match a versioned implementation to the workload

The vLLM 0.28.0 EP guide describes EP width as TP × DP for its documented layout, with attention using its own parallel behavior. These DP ranks can participate in a shared EP group; they are not necessarily independent full-model replicas.

Its backend families include general all-gather/reduce-scatter, DeepEP high-throughput and low-latency paths, and FlashInfer paths for supported multi-node NVLink systems. A backend name is not proof that a particular GPU, quantization kernel, graph mode or topology is supported. Read the exact release's prerequisites and benchmark the relevant shapes.

The DeepEP repository now also documents V2's unified interface and NCCL Gin transport. Historical “prefill backend versus decode backend” advice is therefore version-specific. V2 changes communication buffer requirements too. Record the actual library build and engine integration rather than assuming an old launch line selects the newest path. vLLM Server Arguments That Matter covers preserving that configuration.

Balance work without changing the model's routing semantics

Count expert assignments and time the corresponding kernels per rank and layer. A hot expert can create a straggler, but equal assignment counts need not give equal kernel times. Shapes, hardware, padding and overlap matter. A long collective on an early rank can include waiting for a late peer; it does not uniquely identify a slow network.

The EPLB configuration separates the history window, rearrangement interval and redundant-expert count. Reassignment can improve rank balance without adding copies; replication can split a hot expert's assignments when supported. Preserve the chosen expert function and routing result. Rebalance transfers and monitoring can add overhead even with zero redundant slots.

For uniform per-layer expert storage, incremental redundant-weight memory per rank is approximately:

extra bytes/rank = MoE layers × bytes/expert/layer × extra slots / EP ranks
Synthetic: 60 layers × 40 MB × 8 extra slots / 8 ranks = 2,400 MB/rank
sanity: eight added slots per layer across the group mean one extra slot per rank.

That is 2.4 GB per rank for this example's entire layer stack, not 2.4 GB for an individual expert tensor in one layer. Add scales, alignment, temporary transfers and any uneven placement. Reserve the increment before taking it from KV capacity.

What interviewers are listening for

Choose EP when its measured execution and communication trade meets the service target and memory budget better than supported alternatives. Reverse that choice if sparse per-rank work, exposed exchange time or lost cache capacity erases the gain. There is no universal “expert phase below one third means avoid EP” rule. The baseline phase fraction limits possible benefit, but the actual before/after costs decide.

Key takeaways

• In the synthetic layer, TP and EP both select 80 MB of expert payload across the group; both retain 80 MB per rank.
• The 128-token routing example moves 4 MiB of logical dispatch-plus-return payload per layer before network-specific adjustments.
• Eight extra expert slots per layer across eight ranks add one slot per rank, costing 2.4 GB per rank in the stated 60-layer example.
• Diagnose kernel shape, exposed communication and imbalance separately, with versioned backend support.