hard★ EssentialBasetenModalAnyscale

Design an autoscaler for GPU inference replicas that reacts to load without thrashing.

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsInference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

Advanced

📐 AI Systems Design🔒 Premium

Serverless GPU PlatformsA serverless GPU platform hides allocation and startup from its users. The design balances cold-start latency, tenant isolation and the cost of idle capacity.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

Foundational

🚀 Inference & Serving

Latency Metrics: TTFT, TPOT and GoodputRead a stream from its timestamps: distinguish first-token wait, individual gaps and average generation pace. Compute goodput with explicit deadlines, denominators and failure accounting.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackA new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?Next in this trackWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?Next in this trackHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?

FEDITOR'S NOTE

The control loop must account for startup delay and outstanding launches. Ask how the candidate prevents repeated scale-up and drains a busy replica.

DISCUSSION · 0

No comments yet — be the first to share your approach.