routing

Use a ten-expert example to separate assignments, rank load and dropped routes, then compare balancing controls against quality.

Follow adapter-specific token rows through a shared base, calculate stored adapter bytes, and separate the resident set from the active batch.

A cached replica can still be the slower choice. Compare saved prefill against waiting time, then handle stale cache reports, pending dispatches and tenant boundaries.

An apparent outlier can reflect workload mix, queueing, runtime state or a degraded execution path. Normalize the comparison, run a matched replay and use critical-path arithmetic to test possible causes.

Design a shared catalogue with explicit service tiers, reproducible deployments and bounded cold starts. Residency depends on burst behavior and resource reuse; placement depends on GPU groups and interference as well as memory.

Separate pre-routing from a sequential cascade, charge every attempted generation, and measure checker mistakes. A lower escalation rate is not proof of higher quality, and the largest model is not a correctness oracle.

Attribute queueing, token work and live state before choosing a cap. Bound new admission, verify recovery, and distinguish fair scheduling, reserved service and physical isolation without promising instant or absolute protection.

Group compatible adapters by pinned base, derive their size from target matrices and ranks, and benchmark mixed batches. Catalogue size, CPU residency, GPU slots and active adapters are separate limits; base sharing does not make fifty variants operationally free.

Derive adapter bytes from the target matrices, share compatible base weights, and keep each request's cache identity separate. A worked pool design avoids applying the mixed-adapter cost twice.

Remove the largest region from a concrete capacity plan. Check the surviving budget, tenant eligibility, cold prefixes and retry semantics before claiming the service can fail over.

Work a 200 ms request budget, distinguish stage allowances from percentiles, and price the capacity needed to meet the measured latency contract.

Size both serving pools from workload-matched rates, then check memory, shared transfer paths and surviving capacity. Follow admission and buffer ownership through failure and scaling.

Define model aliases, permission-aware routing, response reuse and bounded retries. Trace a request whose cached answer becomes invalid after its source permissions change.

Implement a stable hash ring, test which keys move, and distinguish balanced ownership from balanced serving traffic.