AI Infra Interviews logo
Kubernetes, Slurm & GPU Scheduling / 16
mediumNewModalBaseten

Our GPU pods take four minutes to start on a fresh node and sometimes fail with 'no CUDA-capable device'. Walk me through both.

Four minutes is a 15 GB image pulled through one registry connection; the CUDA error is a container runtime newer than the host driver. The pull arithmetic layer by layer, what a cache hit actually saves, the compatibility rule with its two exceptions, and the image layout that makes both problems go away.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

Four minutes is a 15 GB image pulled through one registry connection; the CUDA error is a container runtime newer than the host driver. The pull arithmetic layer by layer, what a cache hit actually saves, the compatibility rule with its two exceptions, and the image layout that makes both problems go away.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced
🗂️ Scheduling & Orchestration🔒 Premium
Containers, Images and GPU Cold StartsA GPU container is a 10 to 20 GB image whose CUDA libraries must match a host driver it did not ship with, that loads tens to hundreds of gigabytes of weights before it does anything, and that then spends a minute compiling and warming before the first request is fast. Every one of those steps is a cold-start cost, and the difference between a naive deployment (minutes) and a tuned one (seconds) is a chain of specific fixes: lazy image loading, driver compatibility done right, local weight caches, and snapshots of an initialized process. This page walks the chain with numbers.
Foundational
🩺 Fleet Reliability & Observability
GPU Failure Modes and XID ErrorsWhen a GPU misbehaves, the NVIDIA driver writes an XID line to the kernel log, and the number on that line is the first and often the only clue to what happened. Fleet engineers learn a dozen of them the way doctors learn a dozen lab values: 13 and 31 are almost always the application, 48 and 95 are memory that needs a reset, 63 and 64 are the row remapper reporting or failing, 74 is the NVLink fabric, 79 is a GPU that has vanished from the PCIe bus. This page gives the taxonomy, the decision for each (retry, reset, drain, RMA), and the derivation of how often a big fleet should expect each.
Advanced
🚀 Inference & Serving🔒 Premium
Inference Autoscaling and Cold StartsScaling an LLM fleet is harder than scaling a web service because a replica takes minutes to become useful (pull an image, load 141 GB of weights, warm the cache) and costs several dollars an hour while idle. The signals that work are queue depth and TTFT against the SLO, not GPU utilization, which is misleading for memory-bound decode. The design is a warm pool sized for the burst, hysteresis so the fleet does not thrash, and a cold-start path measured in seconds through snapshots and weight streaming.
Advanced
📐 AI Systems Design🔒 Premium
Serverless GPU PlatformsA serverless GPU platform lets a customer deploy a function or a model and pay only while it runs, so the platform has to start a GPU workload in seconds, pack many customers onto shared hardware without letting them see each other, and keep enough capacity warm that a burst does not wait for a cold start. Each is a design problem with numbers: the cold-start chain and the snapshot that shortens it, bin-packing memory-sized workloads onto fixed-size GPUs, the isolation boundary and its cost, and the economics of idle capacity against cold starts. This page designs the platform and derives the trade-offs.
UP NEXT ON YOUR JOURNEY
FEDITOR'S NOTE

Scored on splitting the pull into layers with a size and a cache status each, on stating the driver-runtime rule precisely (forward compatibility and minor-version compatibility are the two exceptions), and on putting weights outside the image.

DISCUSSION · 0

No comments yet — be the first to share your approach.