TL;DR: Make the comparison equivalent, then inspect per-rank execution and communication. EP can lose because its local expert batches are too small, dispatch/combine is exposed, routing creates stragglers, or the selected kernel/backend is unsuitable. TP can also exploit sparse expert selection, so EP has no automatic total-experts/top-k read advantage. Measure the limiting path and change one factor: placement, supported backend, local batch shape or balancing. Keep EP only if the resulting service improvement pays for communication, memory and operational costs.
How to approach it
First compare hardware budget, model precision, global work, attention layout and service target. Then inspect every rank's expert assignment counts, kernel durations and exchange timing for the same steps. Separate a late sender from a slow transfer. Quantify one candidate improvement, test it, and close with the condition under which TP remains the better choice.
A strong answer
An expert-parallel (EP) deployment across two nodes is reported 18% slower than a tensor-parallel (TP) deployment inside one node. That comparison changes topology, possibly GPU count and attention/cache layout as well as expert placement. It is a useful deployment result if costs are stated, but cannot attribute the difference to EP alone.
Record the exact checkpoint, quantization and kernels; TP/DP/EP group membership; total scheduled tokens across the EP group; prompt lengths; and offered load. In particular, batch one per DP rank is different from one token across the entire EP group. Expert Parallel and All-to-All Backends explains these layout distinctions.
In expert parallelism, each token's chosen experts may live on other GPUs. Dispatch is the all-to-all exchange that sends tokens to those GPUs; combine is the exchange that brings expert outputs back to the token's home GPU.
Test competing explanations
| Hypothesis | Evidence to collect | Focused comparison |
|---|---|---|
| Dispatch/combine is expensive | Aligned rank traces, message sizes, link counters and exposed time | Same placement and workload with another supported backend |
| Expert assignments are uneven | Counts per expert/rank for the same layer and step | Reassignment or bounded redundancy with routing semantics preserved |
| Expert kernels are inefficient | Per-expert token counts, padded shapes and kernel duration | Different supported kernel or larger useful local batches |
| One rank is slow for another reason | Clocks, power, device errors, competing work, prior-layer timing | Equivalent rank or placement under the same workload |
| Attention or cache changed too | State placement, attention time and cache-pressure recovery | Hold attention and admission behavior fixed where possible |
A long collective on one rank can include waiting for another rank's expert computation. Sum neither all ranks' durations nor overlapped kernels and call that the step's communication overhead. Use an Nsight Systems trace across the ranks to identify what blocks the next dependency, aligning the timelines before comparing them. Serving Benchmarks That Do Not Lie distinguishes critical-path diagnosis from end-to-end service capacity.
Do not use payload divided by peak bandwidth to dismiss communication
Use a synthetic GLM-shaped calculation, with 75 MoE layers, hidden width 6,144, top-8 and BF16 dispatch and return. It is a simple logical payload model, not a measured GLM network trace:
Logical bytes/token/layer/direction = 6,144 × 8 × 2 = 98,304 B
Dispatch + return across 75 layers = 98,304 × 2 × 75
= 14,745,600 B = 14.7456 MB
Local assignments, shared destinations and quantized dispatch can reduce physical network traffic; metadata and padding can add costs. Map the payload onto actual links. An 800 Gb/s link has a nominal line rate of 100 GB/s, but that does not mean every exchange sustains it, or that all traffic uses one uncongested path. An advertised bidirectional NVLink bandwidth has a different denominator from a one-direction NIC rate.
For an illustrative serialized path, suppose each of the 150 exchanges exposes 20 microseconds of latency, plus all 14.7456 MB traverses a 100 GB/s path with no overlap:
Latency term = 150 × 20 microseconds = 3 ms
Serialization term = 14.7456 MB / 100 GB/s = 0.147456 ms
Combined model = 3.147456 ms
sanity: the latency term is about 20× the payload serialization term.
The topology and overlap assumptions make this a teaching model, not a deployment prediction. It shows why the small serialization number alone cannot rule out communication as the cause. DeepEP's implementation documentation separates throughput-oriented and latency-oriented behavior and now includes a V2 interface; pin the integration being tested.
Quantify imbalance with the correct baseline
In a synthetic balanced baseline, the expert-compute phase occupies 55% of a 10 ms step. Assume its duration scales linearly with the busiest rank's assignment count and the other 45% stays unchanged. A max/mean ratio of 2.5 then gives:
Balanced: expert 5.5 ms + other 4.5 ms = 10 ms
Imbalanced: expert 5.5 × 2.5 + other 4.5 = 18.25 ms
Step increase = 18.25 / 10 - 1 = 82.5%
Throughput ratio at the same work/step = 10 / 18.25 = 54.8%
That inference requires the stated linear-scaling assumption. Weight-dominated expert execution, padding and grouped-kernel efficiency can violate it. Counts alone do not prove a 2.5× phase slowdown. Also specify whether the 55% was measured before or after imbalance; using the wrong baseline changes the answer.
Match the remedy to measured cost
For exposed exchange time, compare supported backend/topology combinations and useful overlap. For kernel shape, change the supported execution path or group batch rather than blindly adding expert copies. For persistent rank imbalance, test reassignment first and add bounded redundancy if a hot expert requires multiple placements. A small observation window can mistake routing noise for a stable hot spot.
The vLLM 0.28.0 EP guide describes EPLB placement and redundancy controls. Calculate the incremental memory per rank as layers × stored bytes per expert per layer × extra slots / EP ranks, with placement and metadata corrections. For synthetic 60 layers, 40 MB/expert and eight extra slots across eight ranks, this is 2.4 GB per rank, not eight times that amount. The slots are spread across the group. Transfers and reduced KV capacity can erase a compute gain.
I would keep the first change that improves repeated service measurements and the suspected critical-path term. Undo the change if its memory, transfer or latency cost outweighs the gain, or if the trace disproves the diagnosis. Then retain the better TP layout or test a different EP arrangement. Neither a max/mean ratio above two nor an expert phase below one third is a universal decision rule. No multi-GPU deployment was executed for these examples.
What interviewers probe next
- "Does TP read all experts?" No. A sparse TP implementation executes shards of selected experts. EP changes placement and communication, not which expert functions the router selects.
- "Why can equal counts still be slow?" Different token grouping, padding, kernel choice or device behavior can produce unequal durations. Compare time as well as counts.
- "How much redundancy?" Start from persistent measured hot spots and a per-rank memory budget. Validate the improvement after rebalance and under changed traffic.
- "What if the backend improves a microbenchmark but not serving?" Its time may be overlapped or off the critical path, or it may consume resources needed by compute. The service result decides.
Common mistakes
- Comparing different node counts without stating the resource and topology change.
- Inferring exchange latency from peak bandwidth alone.
- Calling every long collective a network defect.
- Multiplying a group-wide redundancy count as though every rank received all copies.
Key takeaways
- TP and EP can both exploit sparse expert selection; compare the same workload and budget.
- In the serialized teaching model, 3 ms of exchange latency dominates 0.147 ms of payload time.
- Under explicit linear-scaling assumptions, a 2.5× expert-phase imbalance grows a 10 ms step to 18.25 ms.
- Profile the critical path, then pay only for the remedy whose measured benefit exceeds its memory and communication cost.