TL;DR: Four tables: Account (team, guaranteed quota in GPUs, borrow limit, priority budget), Balance (credits in GPU-seconds, refilled at the quota rate, decayed usage for fair share), Allocation (job to a set of GPUs with start and end, the thing that debits), Ledger (append-only debits and credits with an idempotency key). Jobs are admitted while the account has balance or is under its guaranteed quota; over-quota jobs run as preemptible borrowers charged at a lower rate; when an entitled job cannot fit, the scheduler evicts the borrower with the highest usage-over-quota ratio. Charging is per second from the allocation record, reconciled nightly against the GPU telemetry.
How to approach it
Ask what a credit buys (a GPU-second, on which GPU class), who grants credits, whether the cluster is fixed or elastic, and whether fairness or utilization wins when they conflict. Say the system is a ledger plus a scheduler policy and draw the data model before any flow. Then walk one job through submit, admit, run, preempt and settle, showing which table changes at each step. Close with the fairness arithmetic and the failure modes of the ledger.
A strong answer
A typical situation: a research organization with 4,096 H100s across 12 teams. Leadership wants each team to have a guaranteed slice, idle capacity to be usable by anyone, and a way to say that one team's launch matters more this month than another's experiments.
Entitlement versus usage. The account holds what a team is owed (guaranteed GPUs, a borrow limit above that, a priority budget it can spend to jump the queue). The balance holds what it has (credits, refilled at the quota rate, and decayed usage that drives fair share). Separating the two is what makes "you are under quota but have no credits" and "you have credits but the cluster is full" expressible; a single number cannot.
refill and spend
team A: guaranteed 256 GPUs → refill = 256 GPU-seconds per second, cap = one week of quota = 256 × 604,800 ≈ 1.55e8 GPU-seconds
a 512-GPU run for 24 h costs 512 × 86,400 = 4.4e7 GPU-seconds
A's balance covers it if A has been idle for 4.4e7 ÷ 256 ≈ 172,000 s ≈ 2 days
otherwise the run is 256 guaranteed + 256 borrowed; the borrowed half is preemptible and charged at 0.5×
sanity: a team that has not used its slice for a week can run 7× its quota for a day; the cap is what
stops a team hoarding a month and then taking the cluster
The debit path. The scheduler writes an Allocation when a gang is placed. A metering loop debits the ledger every 60 seconds: delta = GPUs × 60 × rate, keyed by (allocation, minute) so a retry cannot double-charge. When the job ends, the final partial minute settles. The ledger is append-only; the balance is a cached sum, rebuilt from the ledger if they disagree. Nightly reconciliation compares each allocation's GPU-seconds against DCGM telemetry for the same GPUs; a discrepancy over 2% opens a ticket rather than silently correcting either side.
Fairness. Queues are ordered by decayed usage divided by quota, lowest first, with a half-life of a few days so last month's spending does not block this week's. GPU Job Scheduler Design describes the loop this policy plugs into: fair-share order, head job, all-or-nothing placement, preemption of borrowers when an entitled job cannot fit.
preemption order when team B's entitled 64-GPU job cannot fit
candidates: borrowed gangs only (never a job inside its own guarantee)
score = usage ÷ quota, highest first; tie-break by freshest checkpoint (cheapest to evict)
cost to the victim = T/2 + R: checkpoint every 10 min, restart 3 min → ~8 min lost, credited back
cap: no job preempted more than twice per day; after that it is protected until it finishes
Priorities. A priority budget is a separate scarce currency: spending it moves a job ahead in its own queue and lets it preempt borrowers sooner, and it refills slowly (say 100 points per week). Without a budget, every job is marked urgent by Friday afternoon.
The trade-off to commit to: charge borrowed time at a discount (0.5×) and make it preemptible, rather than charging full price and guaranteeing it. The discount is what fills idle GPUs, since a team will not spend full-price credits on preemptible capacity. The reversal condition: if utilization is already above 90% and the queue is deep, borrowing is rare and the discount only complicates the ledger; charge one rate and drop the borrow tier. The GPU Credit Scheduler Pattern is the same ledger as a coding problem.
Failure modes to name: the double-charge on a metering retry (idempotency key); a scheduler that admits from a stale balance (read the balance in the same transaction as the allocation write); a team that games decayed usage by splitting into sub-accounts (hierarchical accounts with the parent's quota as the cap); credits that expire and cause an end-of-week rush (a rolling cap, not a calendar reset); telemetry drift that undercounts a stuck job (reconcile, do not trust the allocation alone).
What interviewers probe next
- "A team has credits but the cluster is full; what do they see?" A queue position and an estimated start from the fair-share order, plus the option to spend priority budget; credits buy the right to run, not a GPU this second.
- "How do you charge for a job that dies on a bad node?" Refund the last checkpoint interval and restart time from the ledger with a reason code; the failure is the cluster's, not the team's.
- "Why decayed usage and not a monthly reset?" A reset creates a cliff where every team spends on the 30th; decay makes fairness continuous and the cluster load flat.
Common mistakes
- One "credits" column with no separate entitlement, so under-quota and out-of-credit cannot be told apart.
- Debiting from a bill at the end of the job, so a running job's spend is invisible and the account can go deep negative.
- Preempting jobs inside their guarantee.
- No idempotency on the metering write.
Key takeaways
- Account (entitlement), Balance (credits and decayed usage), Allocation (the debit source), Ledger (append-only, idempotent).
- Refill at the quota rate with a rolling cap; borrowed time is discounted and preemptible.
- Fair share = decayed usage ÷ quota; preempt borrowers by that score, freshest checkpoint first, T/2 + R credited back.
- Reconcile the ledger against GPU telemetry nightly and surface discrepancies rather than auto-correct.