TL;DR: Give every team a guaranteed quota it can reclaim by preempting borrowers, let idle quota be borrowed at lower priority, order the queue by fair share (recent usage divided by entitlement, lowest ratio first), and account for every allocated GPU-hour so hoarding costs money. The pool runs near 85% where eight private clusters run near 43%, and the policy holds only if it anticipates hoarding, job splitting, priority inflation and deadline stampedes.
How to approach it
Open by asking what the teams' peak and average demands look like and whether any of them serve production traffic, because a serving team needs a hard wall rather than a quota. Then do the pooling arithmetic first, so the interviewer hears why a shared pool exists before hearing how it is governed. Walk the policy in the order a job meets it: quota, borrowing, fair share, preemption, accounting. Close with the gaming moves, because a policy that has not named them will meet them within a quarter.
A strong answer
A typical situation: a lab has eight research teams, each convinced it needs 128 GPUs at peak, and a finance team asking why utilization on the existing per-team clusters sits under half. The pooling case is arithmetic:
inputs: 8 teams, peak demand 128 GPUs each, average demand 55 GPUs each
private clusters:
bought = 8 × 128 = 1,024 GPUs
used = 8 × 55 = 440 GPUs on average
util = 440 ÷ 1,024 = 43%
one shared pool of 1,024:
the eight peaks rarely coincide, so the pool's peak is closer to 600 to 700 than 1,024
at an 85% target: 0.85 × 1,024 ≈ 870 GPUs of useful work per hour, against 440
sanity: 870 ÷ 440 ≈ 2x the useful work from the same hardware, which is the whole
reason a platform team exists; the price is the policy below and its tickets
The gain vanishes if peaks correlate (every team launches before the same conference) or if one team can sit on capacity it does not use. The policy is what prevents both, and Multi-Tenancy, Quotas and Fair Share covers the mechanisms; here is how they fit together.
Quota is the guarantee. Team A's nominal 128 GPUs are A's whenever A asks, enforced by preempting whoever is borrowing them. In Kueue this is a ClusterQueue's nominal quota inside a cohort; in Slurm it is a QoS or partition limit per account. Hierarchical quotas matter at lab scale: the pretraining organization gets 512, its three subteams split them, and a subteam can borrow from a sibling before borrowing from another organization.
Borrowing is the utilization. When A is idle, B runs on A's 128 at a priority that marks the capacity as borrowed and therefore reclaimable. Without borrowing, quotas rebuild the private clusters inside the pool.
Fair share is the queue order, and it is a ratio, not a rank:
fair-share ratio = decayed usage over the window ÷ entitlement
window: 7 days, half-life 3 days, entitlement = quota share of the pool
team A: quota 128 (12.5% of pool), used 400 GPUs average this week → 400 ÷ 128 = 3.1
team B: quota 128, used 50 → 50 ÷ 128 = 0.39
both submit at once: B goes first, A's job waits even though A is under quota right now
sanity: A ran 3x its entitlement all week by borrowing; the ratio pulls it back toward
12.5% over the half-life without anyone filing a ticket
Preemption is the enforcement. When A submits and its quota is occupied by B's borrowed jobs, the scheduler evicts B's lowest-priority whole gang and requeues it. Whole gangs, never a partial eviction, because a gang missing one rank is worth nothing (Gang Scheduling with Kueue and Volcano). The checkpoint cadence of borrowed jobs therefore sets the cost of being preempted, and teams running on borrowed capacity should checkpoint the way spot users do.
Accounting closes the loop: GPU-hours allocated, per team, per job, per partition, priced, and visible to the team lead. It feeds the fair-share score and the internal bill.
Now the gaming, which is where the answer earns its score:
| Move | What the team does | Fix |
|---|---|---|
| hoarding | placeholder jobs hold quota "in case" | charge allocated hours, used or idle; publish per-team utilization |
| job splitting | one 256-GPU gang becomes 32 jobs of 8 to dodge a per-job cap and to be harder to preempt | caps per team, not per job; preempt by team priority |
| priority inflation | everything is urgent | a fixed monthly budget of high-priority GPU-hours per team |
| deadline stampede | every peak lands the week before a launch | reservations booked ahead and charged whether used or not |
| research steals from serving | borrowing reaches into the serving pool | a hard wall: serving capacity is never borrowable; research borrows only from research |
The decision: quota plus borrowing plus fair share plus preemption, with allocated-hour chargeback. The condition that reverses it toward static quotas is a pool where the teams' peaks are strongly correlated and every team runs at its quota most of the time; there is nothing to borrow, and the preemption churn costs more than it recovers. That is rare, and a team claiming it should be asked for the utilization graph.
The reversal condition: a fleet with one team on it. Quota, borrowing and fair share are all machinery for arbitration, and with no one to arbitrate between they are overhead that slows admission. Multi-Tenancy, Quotas and Fair Share is where the four layers earn their place. kubectl describe resourcequota and the scheduler's pending reasons are where a starved team's evidence comes from.
What interviewers probe next
- "A product team says research is stealing their GPUs. What do you do?" Show that serving capacity sits behind a hard wall and borrowing flows only into research pools; if it does not, that is the bug to fix today.
- "How do you pick the half-life?" Long enough that a weekend of heavy use does not zero a team's priority on Monday, short enough that last month's usage does not matter: 3 to 7 days is where most fair-share configurations land.
- "What stops preemption from thrashing?" A minimum runtime before a borrowed job becomes preemptible, and preempting the most recently started borrower first so the least work is lost.
- "Two jobs want different resource mixes, one GPU-heavy, one CPU-heavy. How do you compare them?" Dominant resource fairness: compare each job on the share of the resource it uses most.
Common mistakes
- Proposing static quotas per team and calling it fair; it is fair and it leaves half the fleet idle.
- Skipping the pooling arithmetic, so the policy sounds like bureaucracy rather than a 2x on useful work.
- Preempting individual pods instead of whole gangs, which leaves the victim job holding GPUs it cannot use.
- Charging for used GPU-hours only, which makes hoarding free.
Key takeaways
- Pooling: 8 × 128 private at 43% used versus one pool at 85%; roughly 2x useful work per GPU.
- Four layers in order: guaranteed quota, borrowing at low priority, fair share as usage ÷ entitlement with a decay window, whole-gang preemption; accounting feeds all of them.
- Charge allocated hours, cap per team, budget priority, charge reservations, wall off serving.
- Kueue cohorts and ClusterQueues, Slurm QoS and fair-share factor, KAI shares are the names to give.