routing

A router that sends 90% of tokens to 26 of 256 experts makes those ranks do nine times their share while everyone else waits. The arithmetic of the imbalance, what a capacity factor drops on the floor, why the auxiliary loss fights the model, and the bias-only balancing DeepSeek-V3 used instead.

Three hundred fine-tunes as three hundred deployments would need three hundred GPUs. Batched adapters put them on one base, and the arithmetic says what each extra adapter costs in memory and in step time.

Least-loaded routing sends a conversation's tenth turn to a replica that has never seen it, and the whole history prefills again. Affinity fixes that and creates hot spots. The router that does both is a scoring function with two terms.

Identical replicas serving identical traffic should produce identical numbers, so a difference is a defect and the fleet median is the tool that finds it. Five causes, the three that are about the replica and the two that are about what it was sent.

Twenty models of different sizes is a packing problem with a cold-start penalty, and the two pull in opposite directions. What stays resident, what loads on demand, the arithmetic that decides which is which, and the interface that stops every team from asking for a dedicated replica.

The cheapest correct answer beats the best answer at scale, so routing is a cost problem with a quality floor. The three routing signals that work, the cascade that trades latency for cost, and the measurement that catches quality loss no latency dashboard shows.

One tenant can consume a shared deployment's capacity without exceeding any limit you set, because the limits are usually on requests and the cost is in tokens. What actually needs limiting, the three isolation levels by strength and price, and the fix that works in minutes.

Fifty full copies is fifty deployments and is unaffordable. Fifty low-rank adapters over one shared base is one deployment, because an adapter is a tiny fraction of the weights and adapters from different tenants can share a batch. The arithmetic that makes it work and the two limits that decide how far it goes.

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.

Two regions at 60% is not one region with a spare: the question is what happens in the 30 seconds after a region drops. Capacity sized for N-1 with the arithmetic, latency routing with a residency override, failover that does not stampede, weights warm everywhere in advance, and the state that must not cross a border.

200 milliseconds at p99 is a budget, and prefill alone spends it on a long prompt. The decomposition into network, queue, routing, prefill and first token, the prompt length the budget allows, chunked prefill in the tail, admission control that rejects what it cannot serve in time, and the utilization the SLO costs.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.

Forty teams calling three providers with a hundred API keys and no idea what it costs is what the gateway exists to end. One endpoint with a model alias layer, routing by policy and health, a cache that is safe only for deterministic calls, fallbacks that degrade rather than fail, budgets per team, and the audit trail.

A plain hash ring with one point per replica measured a 47-fold load imbalance across eight replicas. With 150 virtual nodes each it fell to 1.19-fold, and adding a ninth replica moved 11.9 percent of keys against an ideal of 11.1. Both numbers, and what they cost.