Modal LLM Inference & Serving interview questions

Forecast demand across startup delay, track replicas already being created, and drain slowly enough to avoid repeating the same scale decision.

Trace a model startup through storage, runtime setup and device loading. Transfer floors and dependencies determine whether a minute is plausible.

Thirty percent GPU utilization means kernels ran during about thirty percent of the sampled time. Find out what limits those kernels, what happens between them, and whether requests are meeting their latency targets.

Follow the weight bytes through a small decode batch and a long prefill. A scoped roofline calculation explains the difference without pretending to predict actual latency.

Follow a token into retained attention state, derive bytes per position for a 70B model, and distinguish present cache usage from the room needed to finish a response.

Work from each GPU's cache budget, reserve room for generated tokens, then check latency separately. A limiting-rank example shows why fitting today's prompts is not enough to admit tomorrow's work.

Work from token timestamps to first-token wait, average pace and individual stalls. Then count requests meeting both latency limits, keeping failures and measurement boundaries explicit.

Draw query-head sharing, count actual stored state and compare payloads with explicit model geometry and binary units.

Trace one cache position through a block table, calculate the memory recovered from oversized reservations, and explain why sharing still needs careful writes and matching.

Separate weight storage, matrix arithmetic and attention-cache precision. Choose a supported candidate using task quality, memory and service measurements.

One request finishes, another keeps writing, and a third waits outside. Follow the same workload through two schedules to see what continuous batching improves, and what it cannot promise.

Trace a 20,000-token prompt through a shared token budget, including the last partial chunk. Then choose a setting from measured first-token waits and streaming gaps.

Trace an accepted prefix, a rejection and the correction token; derive emitted progress and compare it with a measured round cost.

Filter engine candidates by exact compatibility, then compare correctness, goodput and operational cost on the same workload.

Build a measured first-token budget and test the smallest serving design that meets it, including cold prompts, queueing and delivery.

Compare a complete phase handoff with a tuned combined pool, then test the queueing reversal and qualified completions. Nominal link speed and fleet size alone cannot decide the split.

Trace the slow requests through queue, engine and gateway clocks. A flat median narrows the investigation, but it does not establish a cause.

Compare proposal methods by committed output over complete round time, with explicit sampling guarantees and target compatibility.