Your expert-parallel deployment is slower than the tensor-parallel one. Find out why.

TL;DR: Three causes and they are distinguishable by measurement. The all-to-all is crossing a slow link, which happens when the expert-parallel group spans nodes and the backend is not matched to the interconnect. The all-to-all backend is wrong for the phase, since multi-node prefill and multi-node decode want different implementations and the default works everywhere without being good anywhere. Or the experts are unevenly selected, so one rank does far more work than the others and the step finishes when it does. The third is the most common and the least measured. Routing is learned rather than uniform, and a rank holding a popular expert can receive several times the mean token count, which turns a well-configured deployment into one that runs at the pace of its busiest rank. Measure the per-rank token distribution before touching anything, since the remedy for imbalance is redundant experts and the remedy for the other two is a flag, and applying the wrong one wastes memory or time.

How to approach it

Establish where the expert-parallel group sits relative to the NVLink domain, because that determines whether the interconnect can even be adequate. Then measure the three candidates rather than reasoning about them: the all-to-all time as a fraction of the step, the backend in use, and the per-rank token distribution. Apply the remedy that matches. Close with the case where expert parallelism is simply the wrong choice.

A strong answer

A typical situation: a 753B mixture-of-experts model is deployed with expert parallelism across two nodes and measures 18 percent lower throughput than the same model with tensor parallelism inside one node. The team's first move is to try each all-to-all backend in turn, which takes a day and finds a small improvement that does not explain the gap.

The first structural question:

where does the expert-parallel group sit?
  inside one NVLink domain (8 GPUs on a node, or up to 72 on a rack-scale system)
    the all-to-all runs at 1.8 TB/s per GPU on Blackwell parts
  across nodes on the scale-out fabric
    it runs at the NIC rate, 800 Gb/s per GPU = 100 GB/s, roughly 18 times slower

what the all-to-all costs, per token per direction
  hidden x bytes x experts per token
  for GLM-5.3's shape: 6,144 x 2 B x 8 = 98,304 B = 98 KB
  twice per MoE layer (dispatch and combine), across 75 sparse layers:
    75 x 2 x 98,304 = 14.7 MB per token

  time for that traffic
    inside a domain at 1.8 TB/s:  14.7e6 / 1.8e12 = 8.2 microseconds
    across the fabric at 100 GB/s: 14.7e6 / 100e9 = 147 microseconds
sanity: 147 microseconds per token against a decode budget of perhaps 20 milliseconds is
        0.7 percent, so a correctly configured cross-node all-to-all is not obviously fatal,
        which means an 18 percent gap is probably not this

Expert Parallel and All-to-All Backends covers the backends and the width formula. NVLink Domains and the NVL72 Rack covers the boundary that the first check is about.

The three causes and how to tell them apart:

the imbalance arithmetic, which is why it dominates
  suppose the busiest rank receives 2.5x the mean token count
  every rank waits for it, so the step takes 2.5x the balanced time for the expert layers
  the other ranks idle for 60 percent of that phase

  what that does to a deployment
    if expert layers are 55 percent of the step, a 2.5x imbalance makes them 137 percent,
      so the step grows by 82 percent
    the fleet delivers roughly half the throughput its bandwidth would allow
    and no backend change moves it, because the traffic was never the problem

how to see it
  enable the balancedness logging that vLLM's expert parallel load balancer provides, or
    instrument the router's output to count tokens per expert over a window
  compare max rank load against mean rank load
    a ratio near 1.1 is healthy
    a ratio above 2 is the answer to this question
sanity: this is measurable in an hour and it is the cause most often left unmeasured, because
        the flags are easier to try than the instrumentation is to add

The remedies, matched to the cause:

interconnect
  move the expert-parallel group inside a domain if the hardware allows
  otherwise reduce the expert-parallel degree so the group fits, accepting more bytes read
    per token in exchange for cheaper communication

backend
  inside an NVLink domain:      an NVLink-specific backend
  multi-node prefill:           the high-throughput DeepEP backend
  multi-node decode:            the low-latency DeepEP backend, which works with CUDA graphs
  unsure or mixed:              the default allgather_reducescatter, which is correct
                                everywhere and best nowhere

imbalance
  --enable-eplb, with --eplb-config controlling the window (1000 engine steps by default),
    the rebalance interval (3000 steps) and the number of redundant experts (0 by default)
  redundant experts replicate hot experts onto additional ranks so their load splits
  the memory price is real: the documentation gives roughly 2.4 GB per redundant expert for
    DeepSeek-V3, so a budget of 8 redundant experts is on the order of 19 GB per rank
sanity: measure first, because enabling the balancer on a balanced workload spends gigabytes
        per rank and gains nothing

The reversal condition: if the model's expert layers are a small fraction of the step, expert parallelism is not worth configuring at all and tensor parallelism is simpler and faster. That happens with a low sparsity ratio, with a small expert intermediate size, or when the deployment is prefill-dominated so the arithmetic rather than the weight reads dominates. The check is to profile one step and see what fraction the mixture-of-experts layers occupy. If it is under about a third, the entire expert-parallel apparatus is optimizing a minority of the time and its communication cost can easily exceed its benefit, which is exactly the situation the question describes.

What interviewers probe next

• "Why does imbalance cost so much?" Because the step is synchronous: every rank waits for the slowest, so the busiest rank sets the pace and the rest idle.
• "How many redundant experts?" As few as the imbalance requires, measured after enabling the balancer, because each one costs gigabytes per rank.
• "Does imbalance change over time?" Yes, with the traffic mix, which is why the balancer uses a rolling window and a rebalance interval rather than a fixed assignment.
• "What if the imbalance is inherent to the workload?" Then redundancy is the answer rather than a fix, and the memory budget for it becomes part of the deployment's sizing.

Common mistakes

• Trying backends before measuring the per-rank token distribution, which is where the answer usually is.
• Enabling the load balancer without measuring imbalance, spending gigabytes per rank for nothing.
• Using the default all-to-all backend on a multi-node decode deployment, where a low-latency implementation belongs.
• Configuring an expert-parallel group that spans the NVLink domain boundary without checking whether the traffic can afford it.
• Applying expert parallelism to a model whose expert layers are a minority of the step.

Key takeaways

• Three causes: the interconnect, the backend, and expert load imbalance, and the third is the most common and least measured.
• All-to-all traffic for a GLM-5.3-shaped model is about 14.7 MB per token, which is 8.2 microseconds inside an NVLink domain and 147 across the fabric.
• A 2.5 times imbalance on expert layers occupying 55 percent of the step grows the step by 82 percent, and no backend change touches it.
• Measure max rank load against mean: near 1.1 is healthy, above 2 is the answer.
• Redundant experts cost roughly 2.4 GB each per rank, so enable the balancer only after measuring the imbalance.