TL;DR: Define each team's entitlement, how unused capacity may be borrowed, when it can be reclaimed and how waiting work is ordered. Borrowing improves utilization only when another team has runnable demand. A quota is not an instant physical reservation, and fairness policy must account for job shape, disruption and starvation.
How to approach it
Start with measured demand and the capacity available for each GPU type. Specify nominal shares and borrowing limits, then the reclaim protocol. Explain the queue ordering separately; resource admission and fair priority are related but different decisions.
A strong answer
For eight teams sharing 1,024 identical GPUs, equal nominal shares would be 1,024/8 = 128 GPUs per team. Use different published weights if the service requires them. Allow borrowing from unused shares only under an explicit reclaim policy, and cap concurrent allocation and pending reservations per tenant. CPU, memory, GPU flavor and topology constraints still apply; 128 GPUs of entitlement is not a guarantee of immediate placement.
To see how borrowing works, simplify the arithmetic to two teams: a 1,000-GPU cluster has two teams with 500-GPU shares. Team A wants 700 GPUs while B wants 200. Strict partitions serve only 700 GPUs even though 200 more could do useful work. Borrowing helps here because demand is waiting. If combined runnable demand were only 440 GPUs, pooling could not turn it into 870 occupied GPUs.
Capacity = 1,000 GPUs; nominal shares A=500, B=500
Demand A=700, B=200
Static served demand = min(700,500) + min(200,500) = 700
Borrowing served demand = 700 + 200 = 900
Gain = 200 GPUs, or 20 percentage points of cluster occupancy
Sanity: neither policy can serve more work than the runnable demand.
Multi-Tenancy, Quotas and Fair Share explains why a maximum, an entitlement and a priority are different. Record which resources the quota covers: GPU model/profile, CPU, memory and any topology constraint. Five hundred free GPUs of an incompatible type do not satisfy a five-hundred-GPU entitlement for another type.
Kueue's ClusterQueue model supports quota and borrowing policies. Its preemption documentation describes configurable ways to reclaim and share resources. The exact fairness algorithm belongs to the deployed implementation; do not describe every system as the same historical-usage ratio.
A policy could order eligible work using weighted recent allocated GPU-hours, with decay, aging and explicit exceptions. Another may compare current borrowed shares. Name the chosen rule, its resource normalization and tie-breaker. Otherwise users cannot predict whether waiting changes their priority. Avoid making low measured utilization free: allocated idle devices still block other work, while utilization telemetry is not a stable billing contract by itself.
Reclaim is a sequence. Identify borrowed allocations, choose victims under the disruption policy, request checkpoint or graceful stop, confirm capacity release, then admit the owner's work. A five-minute reclaim objective must include all those stages plus placement, not just the time to send a termination request. Some jobs are non-preemptible or have large recovery costs; borrowing them may require a reservation window or a different service tier.
In the example, when B's demand rises to 500, A must release 200 to restore both nominal shares. If A's jobs require eight GPUs at a time, victim selection may release more than exactly 200 or fail a topology constraint. Track excess evictions and useful work lost, not just whether the quota ledger balances. Fairness at the queue level does not eliminate fragmentation.
Make the tenant identity stable across users, namespaces and jobs. Otherwise a team can split submissions to obtain several shares. Control who may set priority, and count allocated capacity-time so an idle placeholder is not free. Splitting one long job into many short jobs should not reset the tenant's usage history or create extra entitlement. Bound submission and reservation churn so flooding the queue cannot monopolize scheduler attention. Verify claimed checkpoint capability before lending capacity under a rapid-reclaim promise.
Publish each team's usage, queued demand, borrowing and expected reclaim behavior. Alert on persistent starvation, stale reservations and reclaim-time violations. Replay realistic demand traces before rollout, including a small team returning after a quiet week and a large gang competing with short jobs.
Use borrowing when reclaimable surplus and queued demand make it beneficial. Reverse aggressive borrowing when eviction and restart losses exceed the extra useful work, or when a guaranteed deadline requires reserved capacity. A lower occupancy number can be correct if it buys a service promise users actually need.
What interviewers probe next
- "Does unused quota prove unused hardware?" No; another queue may be borrowing it, or physical constraints may prevent placement.
- "How do small jobs avoid starving a large gang?" Add a tested reservation or aging policy rather than relying on packing alone.
- "What should users see?" Their entitlement, consumption, borrowed amount and reclaim conditions.
Common mistakes
- Claiming pooling increases utilization without additional runnable demand.
- Promising instantaneous reclaim from non-preemptible work.
- Confusing quota admission with physical placement.
Key takeaways
- The example improves occupancy from 700 to 900 by serving 200 queued GPUs of demand.
- Reclaiming a share includes stop, release and placement time.
- Judge useful work and fairness together with occupancy.