all to all

Follow one token to two expert ranks and back, then count logical assignments separately from physical network transfers.

Find whether expert placement, small batches or communication makes the deployment slower. Compare equivalent workloads and inspect the worker that holds up each step.

Track actual values across ranks, then count bytes with an explicit buffer definition. Reduction combines values; gathering and all-to-all change ownership.

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

Size a hypothetical trillion-parameter MoE from its weight placement, attention-state layout and expert traffic. Use rack specifications as bounds, then measure the serving workload before promising throughput.

Separate total and active parameters, then size persistent state, approximate compute and token dispatch. Compare supported parallel layouts using per-rank memory, measured communication and recovery.