CoreWeave Kubernetes, Slurm & GPU Scheduling interview questions

The four layers between `nvidia.com/gpu: 4` and a container that can run CUDA, the one failure each layer produces, the stranded-GPU arithmetic the integer model causes, and why Dynamic Resource Allocation replaces the count with a claim.

Hardware partitions, shared SMs, and context switching are three different promises about isolation and waste. The slice arithmetic for an H100, what each mode gives up, and the one rule about tenants that decides most of it.

Slurm gives you gang scheduling, topology and MPI launch for free and nothing for services; Kubernetes gives you services, elasticity and the ecosystem and makes you bolt on the batch semantics. The hybrid that most large clusters land on, with the numbers that push them there.

A distributed training job is 64 pods that start together or not at all. The deadlock two partially placed jobs produce, counted out on a 64-GPU cluster, and how Kueue and Volcano make the job the unit of admission.

Fractions are either hardware slices with fixed shapes or soft shares with no isolation, and the scheduler has to know which. The representation, the packing score, the fragmentation arithmetic that spread placement produces, and the defragmentation step that fixes it.

Same job, same GPU count, half the speed: the slow one was split 4 and 4 across two nodes and its tensor-parallel all-reduces run at fabric speed instead of NVLink speed. The bandwidth arithmetic, the per-parallelism placement rule, and the scheduler policy that enforces it.

A node with eight healthy H100s that Kubernetes thinks has none. The layered check from kernel driver to container toolkit to device plugin to taints and allocatable, in the order that isolates the fault fastest, with the log lines each layer prints when it is the one that broke.

A GPU node needs a driver, a container toolkit, a device plugin, a metrics exporter and a validator, in that order, and each needs to know which nodes it belongs on. The operator's component chain, the labels NFD writes to trigger it, and the bring-up arithmetic that rules out doing it by hand.

A shared pool recovers the GPUs inference holds for its peaks, but a preempted training gang takes minutes to give them back and an SLO breaks in seconds. The utilization of each design, the reclaim-time arithmetic against the traffic ramp, and the floor-plus-borrow split most fleets land on.

Committed capacity is cheaper per hour and cannot shrink, so the decision is a utilization break-even, not a discount. The arithmetic for where the reserved baseline should sit, why the p50 of steady demand is the target, and how on-demand, spot and capacity blocks absorb everything above it.

Allocated is not usable: a cluster full of half-empty nodes has hundreds of idle GPUs that no gang can take. The arithmetic of how mixed job sizes fragment a fleet, the packing policies that prevent it, the defragmentation moves that repair it, and what each costs the jobs already running.

A pre-flight suite is a list of tests with a number on each: DCGM health, a GEMM near the fleet median, NCCL at rated bandwidth, NICs at line rate, links active, storage reachable. What goes in the two-minute gate, what waits for the long diagnostic, and why the gate pays for itself.

You cannot remove one rank from a synchronous job; you replace the node and restart from checkpoint, and the whole craft is making that take three minutes instead of thirty. The detection signals, the drain-restart-quarantine sequence, the spare-pool arithmetic, and where elastic training changes the answer.

A pod's default network is an overlay that no RDMA NIC will speak. The three ways to give a pod the fabric (host networking, SR-IOV virtual functions via a NIC device plugin, or a macvlan secondary interface), what each costs in isolation and speed, how NCCL finds the right NIC, and the failure that halves bandwidth.

A driver upgrade touches every layer at once: kernel module, CUDA runtime, container toolkit, NCCL, fabric driver, and every job's image. The compatibility matrix that decides whether a job can run on the new node, the canary that proves it, the wave arithmetic for 2,000 nodes, and the rehearsed rollback.

Free GPUs and a pending gang are not a contradiction: the gang needs 64 whole nodes at once and the free GPUs sit on 300 half-used nodes. The isolation order (quota, fit, per-node reasons, one node's device plugin), the command at each step and what it means, and the fixes ranked by displacement.

Static quotas waste half the fleet and a free-for-all starves the small teams. The four-layer policy (guaranteed quota, borrowing, fair-share ordering, preemption) with the pooling arithmetic that justifies it, the fair-share ratio worked by hand, and the five ways teams game it.

A naive cold start for a 7B function is minutes: pull, start, load, initialize. Under a second means deleting links, not speeding them up: lazy image loading, weights from a local cache, a snapshot of the initialized process with its GPU memory, and a warm pool sized from arrival rate. The chain with a number per link.

Containers share a kernel with the code they run, the wrong boundary for code you did not write. gVisor, Firecracker microVMs and full VMs each buy a stronger one at a cost in memory, start time and GPU access. The overhead arithmetic that turns 50,000 sessions into a host count, and the tiered design.

Spot is 60% cheaper until the preemptions eat the discount. The expected-loss formula (waste per preemption = half the checkpoint interval plus restart), the break-even worked at three cadences, the checkpoint cost that pushes back the other way, and the gang arithmetic that says which jobs must never be on spot.

A 100,000-entry queue cannot be scanned every cycle, a big gang cannot wait forever behind small jobs, and a preemption must not discard an hour of work. The loop in order (admit, order, place, reserve, backfill, preempt), the queue-wait arithmetic, and the checkpoint-aware victim rule.

Ray gives you actors, tasks and placement groups inside a cluster of pods that Kubernetes thinks are just pods. Two schedulers, two views of the same GPUs: the idle-node arithmetic, the placement group that cannot be gang-scheduled by the layer below, and the rules that stop the two from wasting each other's capacity.