TL;DR: Find out what the experiment needs before negotiating the GPU count. Show the relevant queue and placement constraints, offer any feasible smaller or later option, and make uncertainty in the start estimate explicit. Use agreed quota and priority rules for conflicts; preempt other work only with the required authority and a recovery plan.
How to approach it
Ask about the experiment, deadline, minimum useful scale and tolerance for interruption. Check eligible capacity and the queue before promising an alternative. Explain the available options and the cost of displacing existing work. Then describe how team-controlled priorities and predictable admission could reduce repeated requests of this kind.
A strong answer
Suppose a researcher asks for 256 GPUs before a paper deadline. The cluster is 94% allocated and four jobs are ahead. Those facts do not yet tell you when this request can start: the free devices might have the wrong accelerator type, be scattered across unsuitable network domains or belong to another quota. GPU Job Scheduler Design and Topology-Aware Scheduling explain why the placement matters as much as the count.
Start with the work the researcher wants to accomplish. A small debugging run may be useful before the full experiment, but a scaling study or a model that needs all 256 GPUs may not have that option. Treat alternatives as questions to test together rather than assuming the initial request is inflated.
| Possible flexibility | Check before offering it |
|---|---|
| Start with fewer GPUs | The model fits and the smaller run answers a useful question |
| Split the work into independent jobs | The experiment permits independent runs; one synchronous job cannot simply be divided |
| Use preemptible capacity | Checkpoint and restart are supported, with an acceptable lost-work budget |
| Reserve a later window | The placement can be made and the estimate includes overruns or failures |
Show the queue information the researcher is authorized to see: their priority, eligible resources, the constraint preventing admission and an estimated start range with its assumptions. A queue position alone may be misleading when jobs have different sizes and topology requirements. Protect other teams’ confidential work while giving enough information to explain the decision.
Working with Researchers covers the collaboration behind this conversation. Explain what you can offer today only after checking it. If nothing useful is available, a clear refusal and an honest estimate are better than an imaginary smaller allocation. Agree on when you will update the estimate, especially if it depends on a running job finishing.
Low GPU utilization deserves investigation but does not by itself authorize reclaiming another team’s reservation. The job could be waiting for data, performing a necessary synchronization or using resources the utilization metric misses. Work with its owner to find the cause and determine whether releasing capacity is safe under the scheduling policy.
For a preemption example, suppose the most recent durable checkpoint is 12 minutes old and restart on the new placement takes eight minutes. At least 20 minutes of progress and recovery time are at stake before any additional queueing. A promise to checkpoint every 20 minutes does not guarantee that every interruption loses less than 20 minutes overall. Verify checkpoint completion, restore time and destination availability.
Multi-Tenancy, Quotas and Fair Share explains how teams can control priorities within their budgets while borrowing idle capacity under known conditions. Borrowing needs a reclamation rule, and gang jobs may have to stop as a whole when part of their allocation is reclaimed. A quota gives a team policy rights; it does not guarantee that a suitable 256-GPU placement exists immediately.
When priorities conflict, identify the role authorized to choose between organizational commitments. Provide the expected delay, recovery cost and alternatives so that person can decide. Preemption may be justified for an agreed higher-priority need, but include notification and safe cleanup. An emergency exception can be legitimate under an established escalation process; record it instead of turning a private favor into an invisible scheduling rule.
What interviewers probe next
- "What if the researcher escalates?" Give the decision owner the constraints, options, costs and uncertainty already discussed.
- "Can the job run on 32 GPUs first?" Only if that configuration fits and produces useful evidence for this experiment.
- "What if the low-utilization job is wasting capacity?" Investigate with its owner and apply the agreed reclamation policy; do not infer permission from one metric.
- "How do you make start estimates useful?" Show a range, the conditions behind it and when it will be updated.
Common mistakes
- Promising a smaller allocation before checking whether it exists or helps.
- Treating total free GPU count as a feasible gang placement.
- Exposing another team’s confidential job details to justify the queue.
- Treating low utilization as permission to terminate a job.
- Promising a deadline without accounting for checkpoint, restart and placement uncertainty.
Key takeaways
- Establish the minimum useful experiment before negotiating the requested size.
- Quota, free capacity and a feasible 256-GPU placement are separate checks.
- A 12-minute-old checkpoint plus an eight-minute restart already costs 20 minutes before queueing.
- Make priorities, authority and estimate uncertainty visible so the decision can be explained.