Kubernetes GPU Scheduling & Slurm Interview Questions

Follow a four-GPU request from advertised capacity to concrete devices, and see why free GPUs across nodes cannot satisfy one pod.

Compare the resource promises of MIG, time-slicing and MPS, then test peak memory and simultaneous demand before packing workloads.

Choose the control plane from job admission, placement, recovery and service needs, with a concrete partial-allocation failure.

See how partial allocation can strand two training jobs. Separate group admission, real placement, readiness and recovery instead of assuming atomic pod startup.

Design fractional GPU admission around explicit shapes and enforceable guarantees, then reconcile reservations with the runtime.

Use actual rank placement and measured job duration to decide whether waiting for locality beats starting immediately.

Separate quota, borrowing, reclaim latency and fairness, with a demand-conserving example of when pooling improves utilization.

Diagnose GPU pod failures by scheduling, container startup and execution boundaries, without treating Allocatable as a health certificate.

Separate discovery labels, GPU resource advertisement and validation, and understand the operator as a reconciler with configurable components.

Define warm, cached and truly cold paths, then size transfer budgets and warm capacity without promising universal sub-second restores.

Choose a sandbox from its threat model and supported workload, then calculate density from total session memory and CPU demand. GPU access needs its own isolation and compatibility review.

Balance durable checkpoint cost against replay and restart loss, with explicit assumptions and a warning sign when the approximation breaks down.

Size usable inference headroom from the complete readiness delay and demand ramp, then decide which training capacity can safely be borrowed.

Build a durable, bounded scheduling loop with reservations and checkpoint-aware preemption, and avoid universal queue-wait formulas.

Trace Kubernetes pod placement and Ray bundle reservation as separate layers, including logical GPU limits and autoscaling delay.

Measure scheduling, image transfer, unpacking and model initialization separately. A missing-device error needs its own checks for allocation, container access and CUDA compatibility.

Compare the cost of committed capacity with the demand each additional GPU would actually serve. Keep billing discounts separate from capacity guarantees, and account for bursts without counting the same work twice.

Count feasible placements for the waiting job, including its per-node and topology requirements. Then compare waiting, reservations and checkpoint/restart consolidation using their actual disruption costs.

DRA gives the scheduler device claims and attributes to work with. Compare that with extended-resource counts, then check the deployed driver and controller support before promising sharing, partitioning or gang placement.

Use a short admission check for recent changes and a deeper acceptance suite for new or repaired nodes. Set thresholds from matched healthy baselines, and make quarantine and return-to-service decisions explicit.

Keep the training state and replace the failed capacity. Trace fault detection, safe checkpoint selection, worker shutdown and restart; size spare capacity from observed failures and repair times.

Follow a GPU request from node inventory through allocation, device confinement and task launch. A single-node, eight-task example shows how resource shape and per-task device numbering fit together.

Make readiness mean that the engine can serve, scale from measured demand and ready capacity, and drain streams before termination. Kubernetes can implement these controls; an operator packages and reconciles them.

Choose the tenant boundary before planning base-model reuse. Account for frozen weights, adapter state and activations separately, then schedule by compatibility, fairness and the real cost of restarting training.

First align the time window and denominator. Then attribute allocated capacity, kernel activity and useful progress to jobs before choosing between reclaim, queue policy and performance work.

Calculate traffic across the actual inter-cloud boundary before deciding. Whole-job placement often avoids repeated transfers, but communication volume, overlap, convergence requirements and available links determine whether a split job is viable.

Size notebook capacity from concurrent demand, memory needs and burst tolerance. Choose a sharing mode for the trust model, and make idle release explicit about which state can actually be restored.

A fabric address, an RDMA device and a working GPU-memory registration path are separate requirements. Verify each inside the pod, then test the actual GPU-to-NIC placement and collective behavior.

Treat the upgrade as a tested change to a compatibility set. Drain safely, canary representative workloads, size waves from usable spare capacity and rehearse restoration of the previous working configuration.

Trace the job from quota reservation through admission to physical placement. Count eligible node shapes, not just free GPUs, and use controller and scheduler evidence to choose the least disruptive fix.