Anyscale Kubernetes, Slurm & GPU Scheduling interview questions

Slurm gives you gang scheduling, topology and MPI launch for free and nothing for services; Kubernetes gives you services, elasticity and the ecosystem and makes you bolt on the batch semantics. The hybrid that most large clusters land on, with the numbers that push them there.

A distributed training job is 64 pods that start together or not at all. The deadlock two partially placed jobs produce, counted out on a 64-GPU cluster, and how Kueue and Volcano make the job the unit of admission.

Spot is 60% cheaper until the preemptions eat the discount. The expected-loss formula (waste per preemption = half the checkpoint interval plus restart), the break-even worked at three cadences, the checkpoint cost that pushes back the other way, and the gang arithmetic that says which jobs must never be on spot.

A 100,000-entry queue cannot be scanned every cycle, a big gang cannot wait forever behind small jobs, and a preemption must not discard an hour of work. The loop in order (admit, order, place, reserve, backfill, preempt), the queue-wait arithmetic, and the checkpoint-aware victim rule.

Ray gives you actors, tasks and placement groups inside a cluster of pods that Kubernetes thinks are just pods. Two schedulers, two views of the same GPUs: the idle-node arithmetic, the placement group that cannot be gang-scheduled by the layer below, and the rules that stop the two from wasting each other's capacity.

The expensive thing is a base model in GPU memory, not the customer's job. Keep bases resident, pack LoRA jobs onto them, isolate tenants at the process and device level, checkpoint adapters in seconds so jobs run on cheap capacity, and attribute cost per tenant. The arithmetic behind each rule.

A synchronous training job across two clouds runs at the speed of the link between them, a thousand times slower than the fabric inside either. The arithmetic that kills the idea for one job, the two cases where it is tolerable, and what a platform does instead: partition the work, not the job, and move data ahead.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.

The four layers between `nvidia.com/gpu: 4` and a container that can run CUDA, the one failure each layer produces, the stranded-GPU arithmetic the integer model causes, and why Dynamic Resource Allocation replaces the count with a claim.

Hardware partitions, shared SMs, and context switching are three different promises about isolation and waste. The slice arithmetic for an H100, what each mode gives up, and the one rule about tenants that decides most of it.

Fractions are either hardware slices with fixed shapes or soft shares with no isolation, and the scheduler has to know which. The representation, the packing score, the fragmentation arithmetic that spread placement produces, and the defragmentation step that fixes it.

Same job, same GPU count, half the speed: the slow one was split 4 and 4 across two nodes and its tensor-parallel all-reduces run at fabric speed instead of NVLink speed. The bandwidth arithmetic, the per-parallelism placement rule, and the scheduler policy that enforces it.

Static quotas waste half the fleet and a free-for-all starves the small teams. The four-layer policy (guaranteed quota, borrowing, fair-share ordering, preemption) with the pooling arithmetic that justifies it, the fair-share ratio worked by hand, and the five ways teams game it.

A node with eight healthy H100s that Kubernetes thinks has none. The layered check from kernel driver to container toolkit to device plugin to taints and allocatable, in the order that isolates the fault fastest, with the log lines each layer prints when it is the one that broke.

A GPU node needs a driver, a container toolkit, a device plugin, a metrics exporter and a validator, in that order, and each needs to know which nodes it belongs on. The operator's component chain, the labels NFD writes to trigger it, and the bring-up arithmetic that rules out doing it by hand.

A naive cold start for a 7B function is minutes: pull, start, load, initialize. Under a second means deleting links, not speeding them up: lazy image loading, weights from a local cache, a snapshot of the initialized process with its GPU memory, and a warm pool sized from arrival rate. The chain with a number per link.

Containers share a kernel with the code they run, the wrong boundary for code you did not write. gVisor, Firecracker microVMs and full VMs each buy a stronger one at a cost in memory, start time and GPU access. The overhead arithmetic that turns 50,000 sessions into a host count, and the tiered design.

A shared pool recovers the GPUs inference holds for its peaks, but a preempted training gang takes minutes to give them back and an SLO breaks in seconds. The utilization of each design, the reclaim-time arithmetic against the traffic ramp, and the floor-plus-borrow split most fleets land on.

Four minutes is a 15 GB image pulled through one registry connection; the CUDA error is a container runtime newer than the host driver. The pull arithmetic layer by layer, what a cache hit actually saves, the compatibility rule with its two exceptions, and the image layout that makes both problems go away.

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

Allocated is not usable: a cluster full of half-empty nodes has hundreds of idle GPUs that no gang can take. The arithmetic of how mixed job sizes fragment a fleet, the packing policies that prevent it, the defragmentation moves that repair it, and what each costs the jobs already running.

Device plugins count GPUs and know nothing else; Dynamic Resource Allocation lets a pod ask for devices by attribute and share them by claim. What the two models can and cannot express, what DRA changes for MIG, NVLink domains and multi-node gangs, and a migration stance for a fleet that runs both.

A pre-flight suite is a list of tests with a number on each: DCGM health, a GEMM near the fleet median, NCCL at rated bandwidth, NICs at line rate, links active, storage reachable. What goes in the two-minute gate, what waits for the long diagnostic, and why the gate pays for itself.