LLM Inference & Serving Interview Questions

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

One request finishes, another keeps writing, and a third waits outside. Follow the same workload through two schedules to see what continuous batching improves, and what it cannot promise.

Trace one cache position through a block table, calculate the memory recovered from oversized reservations, and explain why sharing still needs careful writes and matching.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Trace a 20,000-token prompt through a shared token budget, including the last partial chunk. Then choose a setting from measured first-token waits and streaming gaps.

Trace an accepted prefix, a rejection and the correction token; derive emitted progress and compare it with a measured round cost.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

Separate weight storage, matrix arithmetic and attention-cache precision. Choose a supported candidate using task quality, memory and service measurements.

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

Two requests share 2,000 leading tokens but ask different questions afterward. Follow the shared state through a branch, calculate the reusable blocks, and distinguish memory savings from measured latency savings.

Preserve document identity through packed prompts and completion reordering. Derive rounded cache reservations, then measure feasible wave sizes against the API contract.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Filter engine candidates by exact compatibility, then compare correctness, goodput and operational cost on the same workload.

Follow adapter-specific token rows through a shared base, calculate stored adapter bytes, and separate the resident set from the active batch.

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

Trace a model startup through storage, runtime setup and device loading. Transfer floors and dependencies determine whether a minute is plausible.

Separate checkpoint memory, expert assignments and retained MLA state. The active parameter count alone cannot predict per-rank fit or token latency.

A cached replica can still be the slower choice. Compare saved prefill against waiting time, then handle stale cache reports, pending dispatches and tenant boundaries.

Trace a dropped stream through bounded queues, engine cancellation and safe resource release; define what retries and incomplete delivery mean.

Calculate retained cache and prefill work for a stated 128k model shape, then choose scheduling and memory policies from measured limits.

Compare proposal methods by committed output over complete round time, with explicit sampling guarantees and target compatibility.

Separate device throughput bounds from measured service curves, then choose a feasible operating point using latency, quality and goodput.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

Thirty percent GPU utilization means kernels ran during about thirty percent of the sampled time. Find out what limits those kernels, what happens between them, and whether requests are meeting their latency targets.

Images add preprocessing and encoder work before the language model can answer. Their effect on prompt length and cache memory depends on the model. Budget each stage, and distinguish image-embedding reuse from prefix-cache reuse.

A conversation's cache is leaving GPU memory. Work out whether to save it, reload it later, or recompute, including transfer queues and the chance that nobody returns.

Greedy decoding chooses the highest-scoring token, but identical prompt text does not guarantee identical inputs or numerical execution. Check the effective request and deployment, investigate the first differing token, and define the reproducibility guarantee precisely.

Translate request demand into prompt work, generated tokens and retained state, then size a measured serving configuration and its failure reserve.

Work two audio timelines: input accumulation before transcription and playback starvation after synthesis. Compare real-time factor, response delay and safe concurrent capacity without mixing their units.

Price duplicate work before racing replicas. Separate independent stragglers from shared overload, and define what happens when a winner starts streaming.

Choose the release boundary, account for buffering delay, and keep policy versions attached to decisions. Extend the design to queued review and appeals.

Trace document revisions, cancellation and deadlines so a late correct answer cannot overwrite newer text. Size useful suggestions rather than every keystroke.

Diagnose a stream that begins quickly but produces useful answer content late. Compare generation, parser and client clocks, then test a reasoning budget without hiding truncated or incorrect responses.