141 gigabytes have to move from somewhere to eight GPUs, and every hop has a bandwidth. Add the CUDA init, the engine warm-up and the graph capture, and the minute is gone unless you design each step.
more free answers with an account · no card
← LLM Inference & Serving / 18
hardNewModalBasetenRunPod
A new replica has to load a 70B model and serve traffic in under a minute. Where do the seconds go, and how do you get there?
141 gigabytes have to move from somewhere to eight GPUs, and every hop has a bandwidth. Add the CUDA init, the engine warm-up and the graph capture, and the minute is gone unless you design each step.
Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.
The concepts behind this question
Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.
Advanced
Inference Autoscaling and Cold StartsScaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.🚀 Inference & Serving🔒 Premium
Advanced
Serverless GPU PlatformsA serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.📐 AI Systems Design🔒 Premium
Advanced
Containers, Images and GPU Cold StartsA GPU container is a 10 to 20 GB image whose CUDA libraries must match a host driver it did not ship with, that loads tens to hundreds of gigabytes of weights before it does anything, and that then spends a minute compiling and warming before the first request is fast. Every one of those steps is a cold-start cost, and the difference between a naive deployment (minutes) and a tuned one (seconds) is a chain of specific fixes: lazy image loading, driver compatibility done right, local weight caches, and snapshots of an initialized process. This page walks the chain with numbers.🗂️ Scheduling & Orchestration🔒 Premium
Advanced
Bandwidth-Bound Decode ThroughputBecause decode reads every weight once per step, its speed is a division: memory bandwidth over bytes per step. That one formula gives single-stream tokens per second for any model on any card, the batch curve that flattens at the ridge point, the effect of quantization, and the point where the KV cache rather than the weights becomes the thing being read. This page derives it, works it for a 70B model on four accelerators, and shows how to read a vendor throughput claim against it.🧮 Napkin Math & Capacity🔒 Premium
UP NEXT ON YOUR JOURNEY
Next in this trackWhat changes when you serve DeepSeek-V3 compared with a dense 70B model?Next in this trackHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?Next in this trackYou stream tokens to clients over HTTP. What can go wrong, and how does the server handle each case?
FEDITOR'S NOTE
Scored on tracing the bytes hop by hop with real bandwidths, on separating the weight load from the engine warm-up, and on the tiering that makes the budget achievable without keeping GPUs idle.
DISCUSSION · 0
No comments yet — be the first to share your approach.