Inference Platform Architecture

TL;DR: Seven boxes. A gateway (auth, quotas, per-tenant rate limits, request validation). A router (which model, which replica: prefix-aware so a conversation lands where its KV cache is, load-aware so no replica saturates). A per-replica scheduler inside the engine (continuous batching, chunked prefill, admission when the KV pool is full). Engine replicas (a model copy across one or more GPUs, tensor-parallel within a node). A KV tier (the engine's paged pool on the GPU, with host or remote offload for reuse across replicas in larger designs). An autoscaler (replica count from queue depth and KV pressure, with cold-start-aware warm pools). Observability (TTFT, TPOT, goodput, KV utilization, per-replica health). The sizing: demand in tokens/s ÷ goodput per replica, plus headroom, and the answer for 10,000 concurrent chat users on a 70B model lands near 35 to 45 eight-GPU replicas once prefill and headroom are counted.

The reference shape

Each box has a job and a failure mode, and a design answer that names both for each box is complete.

Box by box

Gateway. Terminates TLS, authenticates the API key, applies per-tenant quotas (requests per minute, tokens per minute, concurrent streams), validates the request (model exists, context fits, parameters sane), and proxies the streamed response. It is stateless and scales horizontally; the failure mode is a rate limiter that is not shared across gateway instances, which lets a tenant exceed its quota by the number of instances (Rate-Limiting Algorithms).

Router. Maps the model name to a pool of replicas, then picks one. The naive choice is round-robin; the choice that matters is prefix-aware: a multi-turn conversation whose earlier turns are cached in replica 7's KV pool should go back to replica 7, because a cache hit turns a 3,000-token prefill into a few hundred and cuts TTFT by most of a second. Combined with load (queue depth, KV pool utilization) so a hot replica does not saturate (Request Routing and Load Balancing for LLMs).

Engine scheduler. Inside vLLM, SGLang or TensorRT-LLM: admits requests into the running batch each iteration, mixes prefill chunks with decode steps so TTFT and TPOT both stay bounded, and refuses or queues when the KV pool is full (Continuous Batching, Chunked Prefill).

Replicas. One model copy each. A 70B in bf16 is 140 GB of weights, so a replica is a tensor-parallel group of 2 to 8 GPUs on one node; a 8B fits on one GPU and a replica is one GPU. Larger designs separate prefill replicas from decode replicas (Disaggregated Prefill and Decode).

KV tier. The paged pool in each replica's GPU memory is the first tier; the design question is whether cached prefixes can be reused across replicas (host memory offload, or a remote KV store on RDMA) so the router's affinity can be looser. At small scale, replica-local plus affinity is enough.

Autoscaler. Sizes the pool from the signals the engines expose (queue depth, KV utilization, TTFT), with the cold-start chain accounted for: a new 70B replica takes a minute or more to serve, so the scaler keeps a warm pool and scales on a leading indicator (Inference Autoscaling).

Observability. Per request: TTFT, TPOT, tokens in and out, cache hit, replica. Per replica: batch size, KV utilization, queue depth, health. Per pool: goodput against the SLO and the error-budget burn (SLOs for AI Systems).

Sizing it, step by step

The derivation an interviewer wants to see, for a concrete workload.

workload: chat product, 10,000 concurrent users at peak, 70B dense model, avg prompt 1,500 tokens
          (of which ~1,000 are a cached prefix on multi-turn), avg answer 300 tokens, SLO p95 TTFT 500 ms, p95 TPOT 50 ms

step 1: token demand
  fraction of users mid-generation at any instant ≈ answer time ÷ (answer time + think-and-read time)
    answer time ≈ 300 × 50 ms = 15 s;  user reads and types ≈ 45 s → ~25% mid-generation
  decode demand ≈ 10,000 × 0.25 × 20 tok/s = 50,000 tok/s
  prefill demand: 10,000 users × (1 request per 60 s) × 500 uncached tokens ≈ 83,000 tok/s of prefill

step 2: goodput per replica (8 × H100, TP8, bf16), from measurement or from the decode formula
  decode: the batch at which p95 TPOT crosses 50 ms is around 64 to 96 → goodput ≈ 1,700 to 2,200 tok/s per replica
  prefill: a replica's prefill rate is compute-bound: 8 × 700 TFLOPS × 40% MFU ÷ (2 × 70e9 FLOPs/token) ≈ 16,000 tok/s
           per replica, and prefill shares the GPUs with decode, so budget ~30% of the replica for it

step 3: replicas
  decode: 50,000 ÷ ~1,900 ≈ 26 replicas if decode had the GPUs to itself
  with prefill taking ~30% of each replica: 26 ÷ 0.7 ≈ 37 replicas
  prefill check: 83,000 ÷ (16,000 × 0.3) ≈ 17 replicas' worth of prefill capacity, under the 37 → fine
  headroom for the peak-hour spike and one failed replica: +20% → ~45 replicas ≈ 360 H100s

step 4: sanity
  360 H100s at $2.50/h ≈ $900/h ≈ $650k/month at peak-sized capacity; with autoscaling to an average of
  half that, ~$330k/month. Serving 10,000 concurrent users at ~$33 per user-month of peak capacity.
  A prefix-cache hit rate that rises from 65% to 85% cuts prefill demand by more than half and drops
  the replica count toward 30; that is why the router's affinity is worth engineering.

Every number in the chain is an assumption the candidate states, and the interviewer's follow-ups are changes to those assumptions: a smaller model (fewer GPUs per replica, more replicas), a coding workload (longer prompts, more prefill), an agent workload (no reader, TPOT relaxed, throughput first).

The failure modes a design must name

• A replica's KV pool fills: admission control queues new requests; without it, the engine preempts running ones and TPOT spikes for everyone (Capacity and Backpressure).
• The router loses affinity (a replica restarts, the hash ring moves): prefill demand jumps as caches miss; the autoscaler must see it as load, not as a fault.
• A slow replica (a throttled GPU): least-loaded routing sends it fewer requests, but the ones it has suffer; outlier detection ejects it.
• Autoscaler lag: a traffic step arrives faster than a replica can cold-start; the warm pool absorbs it or the gateway sheds with a clear status.
• A model rollout: new engine or new weights on a fraction of replicas, with the router splitting traffic and the observability comparing TTFT and quality per version.

Working it in the room

"Design an inference platform for our chat product" wants the seven boxes drawn in the first five minutes, the sizing chain with stated assumptions in the next ten, then the router's affinity and the scheduler's admission as the two deep dives, then the failure modes. The follow-up held back is "traffic doubles in five minutes; what happens?", answered with the autoscaler's lag, the warm pool, and the gateway's shedding with a retry-after. The answer that sounds right and fails is a diagram with a load balancer and "GPU workers" and no numbers: the sizing chain is the answer.

What to remember

• Seven boxes: gateway, router, engine scheduler, replicas, KV tier, autoscaler, observability; each with a job and a failure mode.
• Sizing: demand in tok/s (decode from concurrency × mid-generation fraction × rate; prefill from request rate × uncached tokens) ÷ goodput per replica, adjusted for prefill's share, plus headroom.
• 10,000 concurrent chat users on a 70B lands near 35 to 45 eight-GPU replicas; prefix-cache hit rate is the biggest lever after model size.
• The router's prefix affinity and the engine's admission control are the two deep dives worth the time.
• Name the failure modes: full KV pool, lost affinity, slow replica, autoscaler lag, rollout.