Inference Platform Architecture

TL;DR: Follow a request from the gateway to a replica, then through prefill and decode. Each stage has its own capacity limit. Estimate demand from the workload, measure how much useful work a replica completes within the latency target, and add headroom for bursts, failures and cold starts.

The reference shape

Follow the request through these components. For each one, ask what it owns, what capacity it consumes and what happens when it cannot proceed.

Box by box

Gateway. Terminates TLS, authenticates the API key, applies per-tenant quotas (requests per minute, tokens per minute, concurrent streams), validates the request (model exists, context fits, parameters sane), and proxies the streamed response. It is stateless and scales horizontally; the failure mode is a rate limiter that is not shared across gateway instances, which lets a tenant exceed its quota by the number of instances (Rate-Limiting Algorithms).

Router. Maps the model name to a pool of replicas, then picks one. The naive choice is round-robin; the choice that matters is prefix-aware: a multi-turn conversation whose earlier turns are cached in replica 7's KV pool should go back to replica 7, because a cache hit turns a 3,000-token prefill into a few hundred and saves the time needed to recompute that compatible prefix. The saving depends on the workload and queue. Combined with load (queue depth, KV pool utilization) so a hot replica does not saturate (Request Routing and Load Balancing for LLMs).

Engine scheduler. Inside vLLM, SGLang or TensorRT-LLM: admits requests into the running batch each iteration, mixes prefill chunks with decode steps to balance first-token latency with the pace of active streams, and refuses or queues when the KV pool is full (Continuous Batching, Chunked Prefill).

Replicas. One model copy each. A 70B in bf16 is 140 GB of weights, so choose devices and a sharding layout that fit the weights plus runtime state; a smaller model may fit on one GPU. Larger designs separate prefill replicas from decode replicas (Disaggregated Prefill and Decode).

KV tier. The paged pool in each replica's GPU memory is the first tier; the design question is whether cached prefixes can be reused across replicas (host memory offload, or a remote KV store on RDMA) so the router's affinity can be looser. At small scale, replica-local plus affinity is enough.

Autoscaler. Sizes the pool from the signals the engines expose (queue depth, KV utilization, TTFT), with the cold-start chain accounted for: a new 70B replica takes a minute or more to serve, so the scaler keeps a warm pool and scales on a leading indicator (Inference Autoscaling).

Observability. Per request: TTFT, TPOT, tokens in and out, cache hit, replica. Per replica: batch size, KV utilization, queue depth, health. Per pool: goodput against the SLO and the error-budget burn (SLOs for AI Systems).

Sizing it, step by step

The following is a hypothetical planning exercise. Its service rates, prices and user behavior are assumptions, not benchmark results. A real capacity plan needs a mixed-workload measurement.

workload: chat product, 10,000 concurrent users at peak, 70B dense model, avg prompt 1,500 tokens
          (of which ~1,000 are a cached prefix on multi-turn), avg answer 300 tokens, SLO p95 TTFT 500 ms, p95 TPOT 50 ms

step 1: token demand
  fraction of users mid-generation at any instant ≈ answer time ÷ (answer time + think-and-read time)
    answer time ≈ 300 × 50 ms = 15 s;  user reads and types ≈ 45 s → ~25% mid-generation
  decode demand ≈ 10,000 × 0.25 × 20 tok/s = 50,000 tok/s
  prefill demand: 10,000 users × (1 request per 60 s) × 500 uncached tokens ≈ 83,000 tok/s of prefill

step 2: goodput per replica (8 × H100, TP8, bf16), from measurement or from the decode formula
  decode: the batch at which p95 TPOT crosses 50 ms is around 64 to 96; at a mean TPOT of ~40 ms that is
          64 ÷ 0.040 to 96 ÷ 0.040 ≈ 1,600 to 2,400 tok/s per replica; take ~1,900 as a conservative midpoint
  prefill: a replica's prefill rate is compute-bound: 8 × 700 TFLOPS × 40% MFU ÷ (2 × 70e9 FLOPs/token) ≈ 16,000 tok/s
           per replica, and prefill shares the GPUs with decode, so budget ~30% of the replica for it

step 3: replicas
  decode: 50,000 ÷ ~1,900 ≈ 26 replicas if decode had the GPUs to itself
  with prefill taking ~30% of each replica: 26 ÷ 0.7 ≈ 37 replicas
  prefill check: 83,000 ÷ (16,000 × 0.3) ≈ 17 replicas' worth of prefill capacity, under the 37 → fine
  headroom for the peak-hour spike and one failed replica: +20% → ~45 replicas ≈ 360 H100s

step 4: sanity
  360 H100s at $2.50/h ≈ $900/h ≈ $650k/month at peak-sized capacity; with autoscaling to an average of
  half that, ~$330k/month. Serving 10,000 concurrent users at ~$33 per user-month with autoscaling (≈ $65 if sized for peak).
  If the reused prompt-token fraction rises from 65% to 85%, uncached prompt work falls
  from 35% to 15%. Re-measure the mixed workload before reducing the replica count.

Every number in the chain is an assumption the candidate states, and the interviewer's follow-ups are changes to those assumptions: a smaller model (fewer GPUs per replica, more replicas), a coding workload (longer prompts, more prefill), an agent workload (no reader, TPOT relaxed, throughput first).

The failure modes a design must name

• A replica's KV pool fills: admission control queues new requests; without it, the engine preempts running ones and TPOT spikes for everyone (Capacity and Backpressure).
• The router loses affinity (a replica restarts, the hash ring moves): prefill demand jumps as caches miss; the autoscaler must see it as load, not as a fault.
• A slow replica (a throttled GPU): least-loaded routing sends it fewer requests, but the ones it has suffer; outlier detection ejects it.
• Autoscaler lag: a traffic step arrives faster than a replica can cold-start; the warm pool absorbs it or the gateway sheds with a clear status.
• A model rollout: new engine or new weights on a fraction of replicas, with the router splitting traffic and the observability comparing TTFT and quality per version.

Working it in the room

Start with the product’s traffic and latency requirements. Draw the request path, estimate capacity with stated assumptions, and explain where requests wait or get rejected. Then examine a burst or replica failure: which component detects it, where spare capacity comes from, and what users experience while the system recovers.

What to remember

• Seven boxes: gateway, router, engine scheduler, replicas, KV tier, autoscaler, observability; each with a job and a failure mode.
• Sizing: demand in tok/s (decode from concurrency × mid-generation fraction × rate; prefill from request rate × uncached tokens) ÷ goodput per replica, adjusted for prefill's share, plus headroom.
• The worked assumptions produce about 45 replicas with headroom. User count and model size alone do not determine that number.
• The router's prefix affinity and the engine's admission control are the two deep dives worth the time.
• Name the failure modes: full KV pool, lost affinity, slow replica, autoscaler lag, rollout.