A 64-node gang-scheduled job has been pending for six hours while the cluster shows free GPUs. Debug it live.

Trace the job from quota reservation through admission to physical placement. Count eligible node shapes, not just free GPUs, and use controller and scheduler evidence to choose the least disruptive fix.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Gang Scheduling with Kueue and VolcanoGang scheduling treats a distributed job’s required workers as one admission unit. It prevents partial allocations from holding resources while waiting for the rest of the job.

Core

📐 AI Systems DesignSign in

GPU Job Scheduler DesignA GPU scheduler decides which jobs run, where they run and when resources can be reclaimed. Its placement rules must account for whole jobs, topology and tenant policy.

Foundational

🗂️ Scheduling & Orchestration

Kubernetes GPU SchedulingKubernetes coordinates GPU discovery, scheduling and container access through several components. Understanding their separate roles makes allocation failures easier to diagnose.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Topology-Aware SchedulingTopology-aware scheduling places jobs with the communication network in mind. The right GPU count is not enough when ranks share slow or congested paths.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Closest to this topicWhat is gang scheduling, and what goes wrong on a Kubernetes cluster that does not have it?Closest to this topicHere is a Kubernetes GPU node. Pods requesting nvidia.com/gpu stay Pending. Fix it in front of me and narrate what you check.Closest to this topicOur cluster is 85% allocated but 64-GPU jobs wait for hours. Explain the fragmentation and what a scheduler should do about it.

FEDITOR'S NOTE

Keep quota separate from physical reservations. Use a consistent free-capacity histogram, treat scheduling reasons as evidence rather than unique diagnoses, and avoid expiring health taints without recovery checks.

DISCUSSION · 0

No comments yet — be the first to share your approach.