TL;DR: Separate physical quota, spendable credits, reservations and recorded usage. Admit a job through an atomic reservation and an idempotent allocation command; meter actual allocation intervals into an append-only ledger and settle unused reservation. Fairness chooses who runs when capacity is contested. Credits express a spending policy, not an immediate guarantee that a suitable GPU gang exists.
How to approach it
Define the credit unit and whether GPU classes have different rates. Ask whether balances are hard limits, whether idle quota can be borrowed and what may be preempted. Draw the durable records, then walk a job through reservation, allocation, metering and final settlement. Close with crash recovery and the fairness decision for two competing teams.
A strong answer
Idempotent means repeating a command has the same effect as running it once, so a retried allocation cannot charge an account twice. An account can have money to spend while the cluster has no eight-GPU island available. Another account can be below its nominal quota but out of spendable credits. Model those states separately so the scheduler cannot accidentally treat one as permission for the other.
| Record | Owns |
|---|---|
| Account policy | Resource-class quota, borrowing limits and eligibility |
| Credit ledger | Grants, charges and adjustments with unique event keys |
| Reservation | Amount held for a job, expiry and settlement state |
| Allocation | Resource IDs, owner epoch, start/end and rate version |
| Balance projection | Rebuildable totals and reserved amount used for admission |
Use integer credit units or fixed-point rates rather than floating-point money-like balances. Resource identifiers and rate versions make later reconciliation explainable. GPU Job Scheduler Design handles placement and gang admission; the GPU Credit Scheduler Pattern handles the accounting boundary.
Toy policy: 1 credit buys one allocated GPU-second; no refill during this trace.
Starting balance = 10,000 credits
Reserve four GPUs for at most 600 s: 4 × 600 = 2,400
available = 10,000 − 2,400 = 7,600
Actual allocated duration = 420 s
charge = 4 × 420 = 1,680
release unused reservation = 2,400 − 1,680 = 720
Ending balance = 10,000 − 1,680 = 8,320; reservation = 0
Sanity: available 7,600 + released 720 = 8,320.
Use one transaction to check policy and available credits, create the reservation, and record an allocation intent that can be retried safely. An outbox worker can pass that intent to a separate scheduler. Starting the remote job is outside the database transaction, so reconcile the two sides explicitly. If placement fails, release the hold through an idempotent transition. A job that may outlast its reservation needs renewal or a documented stop or overdraft policy.
Divide allocated time into non-overlapping intervals with stable keys: allocation ID, owner epoch and interval boundaries. Insert a debit and update the balance and reservation projections in the same transaction. A retry then finds the existing charge, and final settlement handles the last partial interval exactly once. After failover, fence the old owner so it cannot keep charging. A uniqueness constraint alone cannot stop two owners from charging the same time under different keys.
Charge allocated time or measured active compute according to the declared contract. If charging allocation time, a low DCGM utilization sample does not erase the allocation. Reconcile scheduler records, node ownership and metering events, preserving discrepancies and reasoned adjustments. Use append-only corrections rather than rewriting historical charges invisibly.
Fairness is a separate scheduling policy. Kueue distinguishes admission and preemption-based fair sharing; no single usage/quota ratio is universal across schedulers. A policy might order by normalized decayed usage, limit borrowing and preempt eligible borrowers when a quota owner needs capacity. Gang constraints mean reclaiming the borrowed portion can stop the whole job unless elastic resizing is supported.
Include recovery in the preemption decision. If a job’s durable checkpoint is two minutes old and restart takes three minutes, stopping it costs roughly five minutes plus any extra queueing or restore work. The expression interval/2 estimates expected checkpoint age under particular event assumptions; here, use the victim’s observed age. Refunds or discounted borrowing are product-policy choices that must be stated before they can be promised.
Choose one simple credit rate and explicit borrowing rules first. Add multiple rates when resource cost or service guarantees justify them. Reverse aggressive preemption when recovery cost destroys useful throughput, but do not make a borrower permanently immune after an arbitrary count while still promising instant quota reclamation. Protect against account splitting with parent-level limits, and show why a job waits: credit, quota, priority or placement.
What interviewers probe next
- "A scheduler crashes after placement?" Reconcile its allocation ID and owner epoch before retrying. The reservation and durable intent should identify the same attempt.
- "A job runs forever?" Require reservation renewal or a hard runtime policy; initial admission cannot authorize unbounded spend.
- "Why keep a balance projection?" Fast admission with transactional consistency; rebuild it from authoritative ledger and reservation records when auditing.
Common mistakes
- Using one number for quota, credits and current availability.
- Double charging overlapping meter intervals with different keys.
- Treating a remote allocation as part of a local SQL transaction.
- Equating allocated GPU time with sampled utilization.
Key takeaways
- The toy reservation of 2,400 settles to 1,680 charged and 720 released.
- Idempotency needs a stable event identity, atomic writes and fenced ownership.
- Fairness, capacity placement and credit accounting answer different questions.