TL;DR: A fractional GPU scheduler needs a typed resource contract, a feasibility check, a reservation protocol and runtime enforcement. A MIG profile, a time-slicing slot and an MPS configuration are different resources. Never promise a memory limit or compute guarantee that the selected backend does not enforce.
How to approach it
Define what the customer receives. Model legal device configurations and job dependencies, reserve a feasible placement, then verify that the runtime created exactly that allocation. Optimize packing only after the hard constraints pass.
A strong answer
A customer asks for half a GPU. (MIG, Multi-Instance GPU, splits a supported GPU into isolated hardware slices; MPS, Multi-Process Service, lets processes share it concurrently; time-slicing lets them take turns.) That could mean half the memory, a throughput target, a spending cap or a scheduling slot. These meanings have different implementations. GPU Job Scheduler Design starts with admission; this design first makes the request unambiguous.
Start with an inventory of the physical GPUs: their stable IDs, models, supported partitions, topology, health and current configuration. Keep this separate from the capacity advertised to customers, since ten shared slots do not mean ten physical devices. Record who owns each active allocation and reservation, which generation it belongs to, and how it expires or recovers. Requests then specify what must fit: peak memory, acceptable sharing mode, tenant restrictions, device count, topology and restart needs.
MIG profiles have device-specific placement rules. Treat the legal layouts as configurations to validate, not a bag of interchangeable gigabytes. Reconfiguring a GPU can disrupt its users; reserve a maintenance transition rather than silently destroying an active profile to improve packing.
For time-slicing, expose a shared-access contract. NVIDIA's sharing documentation does not turn each advertised replica into a fixed memory partition. MPS controls vary with version and mode. If the backend cannot enforce the requested bound, reject or offer a different contract. Scheduler bookkeeping cannot prevent an application from consuming memory that the runtime leaves unrestricted.
Consider a synthetic memory budget; it is not a legal MIG layout:
Usable device memory: 80 GB
Platform reserve: 8 GB
Admissible measured workload peaks: at most 72 GB total
Requests: 28 + 28 + 12 = 68 GB → 4 GB margin after reserve
Another 12 GB request: 80 GB workload + 8 GB reserve = 88 GB → reject
Sanity: calling the last request a small fraction does not create capacity.
Peak measurements need a margin and a defined input envelope. Memory feasibility still says nothing about simultaneous throughput. A set of jobs that fits can violate latency through shared execution, memory bandwidth or host resources. Benchmark the promised co-runner mix and expose the sharing policy to users.
Separate hard constraints from preferences. Health, legal profile, memory contract, tenant boundary and required topology are hard checks. Packing, cache locality and spreading can be preferences. For a multi-device job, reserve the entire feasible set transactionally; do not hold three devices forever while waiting for the fourth. Reservation failure rolls back consistently, and retries use an idempotent allocation identity.
Once the reservation is durable, the node agent applies the device configuration and reports that it is ready. The workload can then start. A timeout needs more care: the agent may have succeeded even though its reply was lost. Check the node's actual state before offering that capacity to another job. A restarted controller must make the same comparison between durable ownership records and running workloads, or it risks assigning one device twice.
Packing has a reversal condition. Filling every device with small long-lived jobs may make large jobs impossible to place. Preserve capacity shapes or use checkpoint-aware defragmentation with a bounded disruption policy. Conversely, strict spreading can waste devices and increase cost. Compare queue wait, completed work, SLO violations and unusable free capacity by request shape, not just average utilization.
What interviewers probe next
- "The controller crashes after configuring a device. What happens?" Reconcile durable ownership with observed runtime state before reallocating it.
- "Can a quota admit an impossible profile?" Yes, if quota and physical feasibility are checked separately and never reconciled.
- "What does preemption require?" A supported stop/recovery protocol and a cost policy, not only deleting a reservation.
Common mistakes
- Treating all fractional backends as equivalent.
- Using a packing score to compensate for a failed hard constraint.
- Releasing ownership before confirming the old workload stopped.
Key takeaways
- In the example, 68 GB of workload leaves 4 GB after an 8 GB reserve.
- Reserve a complete feasible shape, then verify runtime enforcement.
- Reconcile uncertain outcomes before reusing capacity.