Perplexity LLM Inference & Serving interview questions

Q: How should I prepare for the Perplexity LLM Inference & Serving round?

Work the llm inference & serving questions on this page in order, study the underlying concepts linked below, then review Perplexity's full interview process for how this round fits the loop. For Perplexity specifically, interviewers weight this: Serving with a retrieval step inside the latency budget.

Q: Does Perplexity hire AI infrastructure engineers?

Yes, for the inference stack, the GPU fleet and the retrieval and data systems behind the product; check the careers page for current titles.

Q: What does the Perplexity AI infrastructure interview test?

No reliable public breakdown of the loop was found, so we do not list rounds. Prepare serving with a retrieval step inside the latency budget, capacity planning for consumer traffic, and cost per query reasoning.

LLM Inference & Serving is a core part of the Perplexity AI Infrastructure Engineer loop. Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. Below are the llm inference & serving questions to prepare, the ones tagged to Perplexity first, then the highest-signal questions from our LLM Inference & Serving track, each with an answer written to a senior-engineer bar.

WHAT PERPLEXITY LOOKS FOR HERE · Serving with a retrieval step inside the latency budget. See the full Perplexity interview process →

LLM Inference & Serving questions tagged to Perplexity

3 questions · 0 unlocked for you

Open full answer →

More LLM Inference & Serving questions for Perplexity's loop

The highest-signal llm inference & serving questions candidates rate most useful, modeled on what Perplexity's AI Infrastructure Engineer loop tests.

15 questions · 10 unlocked for you

Open full answer →

Concepts behind Perplexity's LLM Inference & Serving round

The vocabulary and mental models these questions assume. Start with the foundations free; the deeper, interview-defining ideas are part of premium.

Foundational

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Foundational

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Core

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🔒 Premium

PagedAttentionPagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Advanced

🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Advanced

🔒 Premium

Speculative DecodingDecode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Advanced

🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🔒 Premium

Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.

Explore the full curriculum →

PERPLEXITY LLM INFERENCE & SERVING FAQ

What LLM Inference & Serving questions does Perplexity ask in interviews?▲

Perplexity's AI Infrastructure Engineer loop draws llm inference & serving questions such as "When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?", "How do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?", "When does offloading the KV cache to CPU memory or NVMe beat recomputing it?". Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. The full set, ordered easy to hard with expert answers, is below.

How should I prepare for the Perplexity LLM Inference & Serving round?▼

Does Perplexity hire AI infrastructure engineers?▼

What does the Perplexity AI infrastructure interview test?▼

Other Perplexity interview rounds

The other tracks Perplexity's AI Infrastructure Engineer loop tests.

Perplexity ai systems design→

Prep the whole Perplexity AI Infrastructure Engineer loop

LLM Inference & Serving is one round. Unlock every answer across Perplexity's full loop, plus the concept curriculum, for 6 months. One payment, no auto-renewal. Free questions in every track to start.

Unlock all answers · ₹2,000 / $25 Full Perplexity interview guide

Independent and not affiliated with Perplexity. All trademarks belong to their owners.

Perplexity LLM Inference & Serving interview questions

LLM Inference & Serving questions tagged to Perplexity

When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?

How do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?

When does offloading the KV cache to CPU memory or NVMe beat recomputing it?

More LLM Inference & Serving questions for Perplexity's loop

Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?

What is the KV cache, and why does it keep growing while a request is being served?

What is the difference between static and continuous batching, and why did it change LLM serving?

How does PagedAttention work, and what problem was it solving?

Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve, and what changes the number?

A long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?

When does speculative decoding speed up serving, and when does it break even or hurt?

Compare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.

You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?

Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?

You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.

Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?

How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

Concepts behind Perplexity's LLM Inference & Serving round

Other Perplexity interview rounds

Prep the whole Perplexity AI Infrastructure Engineer loop