A 64-node gang-scheduled job has been pending for six hours while the cluster shows free GPUs. Debug it live.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Gang Scheduling with Kueue and VolcanoA distributed training job is 64 pods that start together or not at all: if 40 are running and 24 are Pending, the 40 hold their GPUs idle at a collective barrier waiting for ranks that may never come, and two such jobs can deadlock a whole cluster. Gang scheduling makes the job the unit of admission. Kueue and Volcano add queues, quotas, priorities and preemption on top, which is what turns a pile of GPUs into a platform several teams can share without starving each other.

Core

📐 AI Systems DesignSign in

GPU Job Scheduler DesignDesign a scheduler for a shared GPU cluster is the most common design prompt in AI infrastructure interviews, because it touches everything: queues and priorities, gang placement, topology, fairness across teams, preemption and the checkpoints that make it survivable, and the failure handling that keeps a 512-GPU job alive. This page builds the design in layers, states the data model and the scheduling loop, derives the numbers (how long a job waits, how much preemption costs, how much fragmentation wastes), and lists the trade-offs the interviewer will push on.

Foundational

🗂️ Scheduling & Orchestration

Kubernetes GPU SchedulingKubernetes knows nothing about GPUs until something tells it. The NVIDIA device plugin advertises each node's GPUs as a countable resource, the scheduler matches a pod's request to a node with enough of them, and the container runtime wires the device in. That model is enough for one job per GPU and breaks the moment you need sharing, topology or multi-node placement, which is where Dynamic Resource Allocation, the GPU Operator and the batch schedulers come in. Knowing which layer does what is the platform interview's opening question.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Topology-Aware SchedulingTwo placements of the same 64-GPU job can differ by 2x in step time: one keeps every tensor-parallel group on a single NVSwitch node and every data-parallel ring on a single rail, the other scatters ranks across racks and pushes per-layer traffic through the spine. The scheduler is the only thing that can prevent the second placement, because the framework maps ranks to whatever GPUs it is handed. Topology-aware scheduling means the scheduler knows the hierarchy (NVLink domain, rail, rack, spine block) and places gangs to keep traffic low in it.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicWhat is gang scheduling, and what goes wrong on a Kubernetes cluster that does not have it?Closest to this topicHere is a Kubernetes GPU node. Pods requesting nvidia.com/gpu stay Pending. Fix it in front of me and narrate what you check.Closest to this topicOur cluster is 85% allocated but 64-GPU jobs wait for hours. Explain the fragmentation and what a scheduler should do about it.

FEDITOR'S NOTE

Scored on the ordered isolation (quota, fit, per-node reasons, a single broken node) with a command and an interpretation at each step, on reading fragmentation as the usual cause, and on the fixes ranked by cost to other tenants.

DISCUSSION · 0

No comments yet — be the first to share your approach.