hardModalBasetenRunPod

A new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?

Trace a model startup through storage, runtime setup and device loading. Transfer floors and dependencies determine whether a minute is plausible.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace a model startup through storage, runtime setup and device loading. Transfer floors and dependencies determine whether a minute is plausible.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🚀 Inference & Serving🔒 Premium

Inference Autoscaling and Cold StartsInference autoscaling adds or removes model replicas as demand changes. Queue pressure, latency and startup time help determine how much ready capacity to keep.

Advanced

📐 AI Systems Design🔒 Premium

Serverless GPU PlatformsA serverless GPU platform hides allocation and startup from its users. The design balances cold-start latency, tenant isolation and the cost of idle capacity.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Containers, Images and GPU Cold StartsGPU cold starts include scheduling, image loading, model loading and engine initialization. Measure each stage before choosing caches, warm pools or supported snapshot mechanisms.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Bandwidth-Bound Decode ThroughputEstimate decode throughput from bytes per step, then separate aggregate output from per-user pace. A finite-memory example shows how batching, cache reads and quantization change the bound.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?Next in this trackHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?Next in this trackYou stream tokens to clients over HTTP. What can go wrong, and how does the server handle each case?

FEDITOR'S NOTE

Define when the startup clock begins and trace which stages can overlap. Readiness must show that the loaded model revision can serve a real request, not merely that its files arrived.

DISCUSSION · 0

No comments yet — be the first to share your approach.