Control Plane and API Design for GPU Platforms

TL;DR: Make creation idempotent so a retried request does not launch duplicate work. Treat cancellation as an intent that workers must carry out, and reconcile reported execution with desired state. For listings, use a stable ordering and an explicit consistency contract rather than assuming pagination is a snapshot.

Two words from the TL;DR. Idempotent means sending the same request twice has the same effect as sending it once, so a client can retry safely after a timeout. Reconciling means a controller repeatedly compares what should be running with what workers report, and fixes the difference.

The resource model, and what is deliberately not in it

/v1/jobs           a unit of work: image, command, resource request, queue
/v1/queues         a named pool with a quota and a priority band
/v1/nodes          physical capacity, read-mostly, written by the agent
/v1/jobs/{id}/events   an append-only record of what happened to one job

deliberately absent
  the allocation itself, because exposing it invites callers to manage placement
  the scheduler's internal ordering, because publishing it makes it impossible to change
sanity: an API is a promise about what will not change, so the smallest resource model that
        serves the use cases is the one that can still be improved later

The state machine the server enforces

Two rules make the diagram enforceable rather than decorative. Terminal states are absorbing, so a transition out of one is a conflict rather than a silent no-op, which stops a late agent report from resurrecting a finished job. And a transition into the state a job already holds is accepted, because the agent reporting success may retry that report and must not receive an error for doing so.

The three semantics

idempotent creation
  the caller supplies a key; the server scopes it to (owner, key) and stores it with the job
  a replay returns the identical resource, and the status code distinguishes the cases:
    201 on the first request, 200 on the replay
  the uniqueness must be a database constraint, not an application check, or two concurrent
    retries both pass the check and both insert
  choose a retention period covering the promised retry window; an illustrative value is 24 hours and stores a hash of the request
    body, so a reused key with a different body is an error rather than a wrong replay

  what it prevents, priced
    a retried create on a 64-GPU job that runs 18 hours before anyone notices the pair
    64 x 18 x $2.5 = $2,880 for one retry
sanity: the cost of the mechanism is one indexed column and a lookup, so this is not a design
        preference

cancellation as intent
  a running process is on a node, and only that node's agent can stop it
  so the endpoint records intent, returns 202 with the current state, and the caller polls
  cancelling an already-terminal job is success rather than an error, because the caller's
    goal is that the job is not running and it already is not

cursor pagination
  order by a stable total key, such as (created_at, id), because a timestamp alone is not
    total and two jobs can share one
  the cursor encodes the last row's key; the next page is everything after it
  offset pagination breaks here: inserting a job before the cursor shifts every later row, so
    a later page can repeat or skip rows as insertions or deletions shift offsets

Reconciliation, and why the control plane is not the truth

The control plane records what should be true and the agents report what is true, and the two diverge constantly: a node reboots, an agent restarts, a process dies without reporting. A design that treats its own database as authoritative accumulates jobs marked RUNNING that are not.

The pattern that holds is periodic reconciliation. Agents report the set of processes they are actually running, the control plane compares that against its own view, and differences resolve in the agent's favour for observed state and in the control plane's favour for desired state. That is the same loop Kubernetes runs, and adopting it explicitly is cheaper than discovering its necessity through a growing population of phantom jobs.

What interviewers are listening for

Trace a create request whose response is lost, then show how a retry finds the same job. Explain what cancellation acknowledges and how the caller learns that execution has stopped. Finally, consider stale worker reports and concurrent listing changes. These cases reveal whether the API’s promises match what the distributed system can enforce.

Key takeaways

• Four resources: jobs, queues, nodes, events. Terminal job states absorb, and a repeat of the current state is accepted rather than rejected.
• Idempotent creation keyed on (owner, key) with a database uniqueness constraint; one retried create on a 64-GPU job can cost about $2,880.
• Cancellation is intent: return 202, let the node agent act, and treat cancelling a finished job as success.
• Paginate with a cursor over (created_at, id), because offsets skip rows when work is created during a listing.
• Reconcile periodically against what agents report, or the control plane accumulates jobs it believes are running and are not.