mediumNewCoreWeaveTogether AI

Place GPU jobs onto nodes. Compare first fit, best fit and worst fit, and say which one a training cluster wants.

Three one-line policy differences produce different fragmentation, and the metric that separates them is not utilization. The measured outcome on the same job sequence, why the largest free block is what matters on a training cluster, and the case where the intuitive policy is exactly wrong.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

📐 AI Systems DesignSign in

GPU Job Scheduler DesignDesign a scheduler for a shared GPU cluster is the most common design prompt in AI infrastructure interviews, because it touches everything: queues and priorities, gang placement, topology, fairness across teams, preemption and the checkpoints that make it survivable, and the failure handling that keeps a 512-GPU job alive. This page builds the design in layers, states the data model and the scheduling loop, derives the numbers (how long a job waits, how much preemption costs, how much fragmentation wastes), and lists the trade-offs the interviewer will push on.

Advanced

🗂️ Scheduling & Orchestration🔒 Premium

Topology-Aware SchedulingTwo placements of the same 64-GPU job can differ by 2x in step time: one keeps every tensor-parallel group on a single NVSwitch node and every data-parallel ring on a single rail, the other scatters ranks across racks and pushes per-layer traffic through the spine. The scheduler is the only thing that can prevent the second placement, because the framework maps ranks to whatever GPUs it is handed. Topology-aware scheduling means the scheduler knows the hierarchy (NVLink domain, rail, rack, spine block) and places gangs to keep traffic low in it.

Advanced

🔌 Networking & Storage🔒 Premium

Topology-Aware CommunicationThe same collective can run at 900 GB/s or at 50 GB/s depending on which links it is laid across, so the mapping of parallel groups onto hardware is a performance decision, not a deployment detail. The rule: tensor-parallel groups inside the NVLink domain, data-parallel rings along rails, pipeline stages across the fabric, and every rank placed so its partner is one hop away. NCCL discovers the topology and does most of this when the job lets it; the failures come from placements that do not.

Advanced

🚀 Inference & Serving🔒 Premium

PagedAttentionPagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackTokens arrive as byte fragments and a character can span several. Write the decoder that streams text correctly.Next in this trackSimulate a ring all-reduce on arrays. Verify both the result and the bytes each rank sends.Next in this trackImplement the block allocator behind a paged KV cache, including prefix sharing and copy-on-write.

FEDITOR'S NOTE

Scored on measuring the largest contiguous free block rather than total free capacity, on best or first fit for gang scheduling, and on knowing worst fit spreads and destroys whole-node availability.

DISCUSSION · 0

No comments yet — be the first to share your approach.