Control Plane and API Design for GPU Platforms

TL;DR: The control plane is the platform, as far as its users are concerned, so the API's semantics matter more than its resource model. Four resources carry a GPU platform: jobs, queues, nodes and events. A job moves through a state machine the server enforces, with terminal states absorbing so a late agent report cannot resurrect a finished job. Then three semantics. Creation takes a client-supplied idempotency key scoped to the caller, backed by a unique constraint in the database, so a retry after a timeout returns the existing job rather than starting a second sixty-four-GPU run. Cancellation is a request rather than a state the caller sets, because a running process lives on a node and only its agent can stop it, so the endpoint accepts and the caller polls. And listing uses an opaque cursor over a stable total order, because offsets skip rows the moment anything is inserted during the walk.

The resource model, and what is deliberately not in it

/v1/jobs           a unit of work: image, command, resource request, queue
/v1/queues         a named pool with a quota and a priority band
/v1/nodes          physical capacity, read-mostly, written by the agent
/v1/jobs/{id}/events   an append-only record of what happened to one job

deliberately absent
  the allocation itself, because exposing it invites callers to manage placement
  the scheduler's internal ordering, because publishing it makes it impossible to change
sanity: an API is a promise about what will not change, so the smallest resource model that
        serves the use cases is the one that can still be improved later

The state machine the server enforces

Two rules make the diagram enforceable rather than decorative. Terminal states are absorbing, so a transition out of one is a conflict rather than a silent no-op, which stops a late agent report from resurrecting a finished job. And a transition into the state a job already holds is accepted, because the agent reporting success may retry that report and must not receive an error for doing so.

The three semantics

idempotent creation
  the caller supplies a key; the server scopes it to (owner, key) and stores it with the job
  a replay returns the identical resource, and the status code distinguishes the cases:
    201 on the first request, 200 on the replay
  the uniqueness must be a database constraint, not an application check, or two concurrent
    retries both pass the check and both insert
  a real implementation expires keys after about 24 hours and stores a hash of the request
    body, so a reused key with a different body is an error rather than a wrong replay

  what it prevents, priced
    a retried create on a 64-GPU job that runs 18 hours before anyone notices the pair
    64 x 18 x $2.5 = $2,880 for one retry
sanity: the cost of the mechanism is one indexed column and a lookup, so this is not a design
        preference

cancellation as intent
  a running process is on a node, and only that node's agent can stop it
  so the endpoint records intent, returns 202 with the current state, and the caller polls
  cancelling an already-terminal job is success rather than an error, because the caller's
    goal is that the job is not running and it already is not

cursor pagination
  order by a stable total key, such as (created_at, id), because a timestamp alone is not
    total and two jobs can share one
  the cursor encodes the last row's key; the next page is everything after it
  offset pagination breaks here: inserting a job before the cursor shifts every later row, so
    page two with an offset skips a row page one already passed

Reconciliation, and why the control plane is not the truth

The control plane records what should be true and the agents report what is true, and the two diverge constantly: a node reboots, an agent restarts, a process dies without reporting. A design that treats its own database as authoritative accumulates jobs marked RUNNING that are not.

The pattern that holds is periodic reconciliation. Agents report the set of processes they are actually running, the control plane compares that against its own view, and differences resolve in the agent's favour for observed state and in the control plane's favour for desired state. That is the same loop Kubernetes runs, and adopting it explicitly is cheaper than discovering its necessity through a growing population of phantom jobs.

What interviewers are listening for

The three semantics, named, with the failure each prevents. Candidates routinely produce a clean resource model and then have no answer for what a retried create does, which is the question the round is actually built on. The second signal is cancellation modelled as intent, because it shows the candidate knows the control plane cannot stop a process. The third is reconciliation, which separates people who have operated a platform from people who have specified one.

Key takeaways

• Four resources: jobs, queues, nodes, events. Terminal job states absorb, and a repeat of the current state is accepted rather than rejected.
• Idempotent creation keyed on (owner, key) with a database uniqueness constraint; one retried create on a 64-GPU job can cost about $2,880.
• Cancellation is intent: return 202, let the node agent act, and treat cancelling a finished job as success.
• Paginate with a cursor over (created_at, id), because offsets skip rows when work is created during a listing.
• Reconcile periodically against what agents report, or the control plane accumulates jobs it believes are running and are not.