Crusoe Kubernetes, Slurm & GPU Scheduling interview questions

Slurm gives you gang scheduling, topology and MPI launch for free and nothing for services; Kubernetes gives you services, elasticity and the ecosystem and makes you bolt on the batch semantics. The hybrid that most large clusters land on, with the numbers that push them there.

A pre-flight suite is a list of tests with a number on each: DCGM health, a GEMM near the fleet median, NCCL at rated bandwidth, NICs at line rate, links active, storage reachable. What goes in the two-minute gate, what waits for the long diagnostic, and why the gate pays for itself.

Slurm's GPU path is four pieces: GRES declares the devices, a partition groups nodes with limits, the batch script asks for a shape, and cgroups confine the job to its GPUs. The script that gets eight GPUs on one node, the flags that scatter them across eight nodes, and how ranks find their GPU.

A synchronous training job across two clouds runs at the speed of the link between them, a thousand times slower than the fabric inside either. The arithmetic that kills the idea for one job, the two cases where it is tolerable, and what a platform does instead: partition the work, not the job, and move data ahead.

The four layers between `nvidia.com/gpu: 4` and a container that can run CUDA, the one failure each layer produces, the stranded-GPU arithmetic the integer model causes, and why Dynamic Resource Allocation replaces the count with a claim.

Hardware partitions, shared SMs, and context switching are three different promises about isolation and waste. The slice arithmetic for an H100, what each mode gives up, and the one rule about tenants that decides most of it.

A distributed training job is 64 pods that start together or not at all. The deadlock two partially placed jobs produce, counted out on a 64-GPU cluster, and how Kueue and Volcano make the job the unit of admission.

Fractions are either hardware slices with fixed shapes or soft shares with no isolation, and the scheduler has to know which. The representation, the packing score, the fragmentation arithmetic that spread placement produces, and the defragmentation step that fixes it.

Same job, same GPU count, half the speed: the slow one was split 4 and 4 across two nodes and its tensor-parallel all-reduces run at fabric speed instead of NVLink speed. The bandwidth arithmetic, the per-parallelism placement rule, and the scheduler policy that enforces it.

Static quotas waste half the fleet and a free-for-all starves the small teams. The four-layer policy (guaranteed quota, borrowing, fair-share ordering, preemption) with the pooling arithmetic that justifies it, the fair-share ratio worked by hand, and the five ways teams game it.

A node with eight healthy H100s that Kubernetes thinks has none. The layered check from kernel driver to container toolkit to device plugin to taints and allocatable, in the order that isolates the fault fastest, with the log lines each layer prints when it is the one that broke.

A GPU node needs a driver, a container toolkit, a device plugin, a metrics exporter and a validator, in that order, and each needs to know which nodes it belongs on. The operator's component chain, the labels NFD writes to trigger it, and the bring-up arithmetic that rules out doing it by hand.

A naive cold start for a 7B function is minutes: pull, start, load, initialize. Under a second means deleting links, not speeding them up: lazy image loading, weights from a local cache, a snapshot of the initialized process with its GPU memory, and a warm pool sized from arrival rate. The chain with a number per link.

Containers share a kernel with the code they run, the wrong boundary for code you did not write. gVisor, Firecracker microVMs and full VMs each buy a stronger one at a cost in memory, start time and GPU access. The overhead arithmetic that turns 50,000 sessions into a host count, and the tiered design.

Spot is 60% cheaper until the preemptions eat the discount. The expected-loss formula (waste per preemption = half the checkpoint interval plus restart), the break-even worked at three cadences, the checkpoint cost that pushes back the other way, and the gang arithmetic that says which jobs must never be on spot.

A shared pool recovers the GPUs inference holds for its peaks, but a preempted training gang takes minutes to give them back and an SLO breaks in seconds. The utilization of each design, the reclaim-time arithmetic against the traffic ramp, and the floor-plus-borrow split most fleets land on.

A 100,000-entry queue cannot be scanned every cycle, a big gang cannot wait forever behind small jobs, and a preemption must not discard an hour of work. The loop in order (admit, order, place, reserve, backfill, preempt), the queue-wait arithmetic, and the checkpoint-aware victim rule.

Ray gives you actors, tasks and placement groups inside a cluster of pods that Kubernetes thinks are just pods. Two schedulers, two views of the same GPUs: the idle-node arithmetic, the placement group that cannot be gang-scheduled by the layer below, and the rules that stop the two from wasting each other's capacity.

Four minutes is a 15 GB image pulled through one registry connection; the CUDA error is a container runtime newer than the host driver. The pull arithmetic layer by layer, what a cache hit actually saves, the compatibility rule with its two exceptions, and the image layout that makes both problems go away.