Multi-Region Serving and Failover
Multi-region serving can improve geographic reach and survive a regional outage. It requires enough remaining capacity and a clear plan for state, routing and degraded service.
TL;DR: Decide which failures the service must survive and how much traffic the remaining regions can absorb. Replicate required model artifacts and configuration ahead of time. Account for cold caches during failover, and choose an explicit degraded mode when spare capacity cannot meet the normal latency target.
The headroom arithmetic, which decides the shape
R equal-capacity regions with evenly balanced load, each at utilization u, one region fails
the failed region's load is redistributed across the remaining R - 1
each survivor now carries u + u/(R-1) = u x R/(R-1)
surviving requires u x R/(R-1) <= 1, so u <= (R-1)/R
R = 2: u <= 0.50
R = 3: u <= 0.67
R = 4: u <= 0.75
R = 6: u <= 0.83
what that costs
two-region active-active pays for twice the capacity it uses, since each region idles half
four regions pay for 1.33 times, which is why large active-active deployments have several
regions rather than two
sanity: this is the same N+1 argument as replicas within a region, applied one level up, and
it is the reason a two-region design is often better run active-passive than
active-active
What crosses a region and what does not
Place the artifacts needed for recovery in the destination before an outage. That includes the intended model version, tokenizer and configuration. Decide separately how usage accounting, audit records and quota enforcement remain correct when regions are unavailable.
KV-cache transfer is a performance choice, subject to the data’s allowed locations and engine compatibility. Recomputing a prefix may be cheaper than shipping its state, but the answer depends on context size, bandwidth and available compute. Session affinity improves reuse; losing it should cause recomputation or a controlled failure, not incorrect output.
For scale, a hypothetical 2,000-position cache at 320 KiB per position contains about 655 MB. Compare transfer time at an achievable cross-region rate with measured prefill time and the downstream queue. Neither option has a universal advantage.
The failover itself
Regional health combines errors, latency and available capacity. Use sustained signals to avoid repeated traffic moves.
No
- Continue normal routing
Use proximity, load and the current health policy.
Yes
- Shift toward healthy regions
For equal regions and balanced traffic, survivor utilization becomes u × R / (R − 1).
- Account for cache misses
Lost prefix affinity adds prefill work during recovery.
- Enforce the degraded-mode policy
Shed or bound waiting when survivors cannot meet the target.
Two behaviours have to be designed rather than assumed. Hysteresis, because a region that flaps in and out moves traffic repeatedly and each move costs the cache affinity of every session it carries. And graceful degradation, because the honest outcome when survivors cannot hold the objective is shedding or queueing with a stated wait rather than accepting everything and missing the target for everyone.
When one region is enough
Multi-region is frequently premature. It doubles the operational surface, requires the weight replication and configuration discipline above, and buys nothing if the product's users are concentrated in one place and the availability target is met by replica-level redundancy within a region. The questions that decide it are where the users are, what the availability target actually is in numbers, and whether a regional failure is a scenario the business has priced. Building two regions to raise availability that was already sufficient is a common and expensive way to make a platform harder to operate.
What interviewers are listening for
Derive the spare capacity for the failure scenario using stated assumptions about region size and traffic balance. Explain how users reach a healthy region, which state is already available there, and how cache misses affect the recovery load. Choose cache recomputation or transfer from measured cost rather than a universal rule.
Key takeaways
- Surviving one region failure requires each region below (R-1)/R utilization: 50 percent at two regions, 75 at four.
- Replicate required weights and configuration ahead of time. Compare cache transfer with recomputation using the actual path and recovery budget.
- Session affinity to a region is what keeps prefix caches useful, and a failover accepts losing it.
- Design hysteresis into the health signal, because a flapping region costs the cache affinity of every session it moves.
- Decide the degraded mode in advance: shed or queue with a stated wait rather than missing the objective for everyone.