Databricks LLM Inference & Serving interview questions

Q: How should I prepare for the Databricks LLM Inference & Serving round?

Work the llm inference & serving questions on this page in order, study the underlying concepts linked below, then review Databricks's full interview process for how this round fits the loop. For Databricks specifically, interviewers weight this: Model serving, fine-tuning and vector search on the platform.

Q: Does Databricks hire AI infrastructure engineers?

Yes: Senior Machine Learning Engineer, GenAI Platform (Mosaic AI: serving, fine-tuning, vector search) in San Francisco and platform infrastructure roles, plus roles in its Bengaluru engineering centre; per 2026 postings.

Q: What does the Databricks AI infrastructure interview test?

Databricks publishes an official interview preparation guide for the general loop. For the platform roles, prepare backend system design with GPUs as the scheduled resource, multi-tenant serving and fine-tuning service design, and training data pipelines; we found no AI-infra-specific first-hand debrief and do not list unconfirmed rounds.

LLM Inference & Serving is a core part of the Databricks ML Platform Engineer loop. Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. Below are the llm inference & serving questions to prepare, the ones tagged to Databricks first, then the highest-signal questions from our LLM Inference & Serving track, each with an answer written to a senior-engineer bar.

WHAT DATABRICKS LOOKS FOR HERE · Model serving, fine-tuning and vector search on the platform. See the full Databricks interview process →

LLM Inference & Serving questions tagged to Databricks

1 questions · 0 unlocked for you

Open full answer →

More LLM Inference & Serving questions for Databricks's loop

The highest-signal llm inference & serving questions candidates rate most useful, modeled on what Databricks's ML Platform Engineer loop tests.

15 questions · 10 unlocked for you

Open full answer →

Concepts behind Databricks's LLM Inference & Serving round

The vocabulary and mental models these questions assume. Start with the foundations free; the deeper, interview-defining ideas are part of premium.

Foundational

Prefill vs DecodeAn LLM request runs in two phases with opposite hardware profiles: prefill reads the whole prompt in one compute-bound pass and decides time to first token, decode emits one token per forward pass and is bound by memory bandwidth. Every serving decision, from batch size to which GPU to buy to whether to split the two phases across machines, follows from that split.

Foundational

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Core

Continuous BatchingContinuous batching schedules at the granularity of a single decode step instead of a whole request, so a finished sequence's slot is refilled on the next iteration rather than when the longest request in the batch ends. It is the scheduling idea that turned LLM serving from a padded, half-idle GPU into one that stays full, and it decides how the engine's scheduler, memory manager and latency SLOs interact.

Advanced

🔒 Premium

PagedAttentionPagedAttention stores the KV cache in fixed-size blocks scattered across HBM and maps each sequence's logical positions to physical blocks through a block table, the same trick an operating system uses for virtual memory. It removes the reservation and fragmentation waste of contiguous allocation, lets blocks be shared between sequences, and is why an engine can decide admission by counting free blocks.

Advanced

🔒 Premium

Chunked PrefillA long prompt's prefill can occupy a GPU for hundreds of milliseconds, and every sequence mid-decode on that GPU waits for it. Chunked prefill splits the prompt into fixed token budgets and interleaves each chunk with a decode step, so decode latency stays flat at the cost of a slower first token for the long prompt. The chunk budget is a knob between TTFT and TPOT, and the interview question is how you would set it.

Advanced

🔒 Premium

Speculative DecodingDecode is memory-bound: each step reads every weight to produce one token. Speculative decoding has a cheap draft propose several tokens, then verifies them all in one forward pass of the big model, so one weight read yields several tokens with output distribution unchanged. It wins 2x to 3x at small batch, breaks even near the ridge point where the GPU is already compute-bound, and lives or dies on the acceptance rate, which is what interviewers ask you to reason about.

Advanced

🔒 Premium

Disaggregated Prefill and DecodePrefill is compute-bound and decode is memory-bound, so running both on the same GPUs means each phase interferes with the other and neither runs on the hardware it wants. Disaggregation puts them on separate pools and ships the KV cache from prefill nodes to decode nodes over the fabric. It lets TTFT and TPOT scale independently and puts high-bandwidth parts where they pay, at the price of a KV transfer per request and a control plane. It pays at scale with long prompts; it does not pay for a small fleet.

Advanced

🔒 Premium

Prefix Caching and KV ReuseMost requests to a production LLM share a prefix: the same system prompt, the same few-shot examples, the same conversation up to the latest turn. Prefix caching keeps the KV blocks for those tokens resident and skips their prefill, so a 4,000-token system prompt costs compute once instead of once per request. Radix trees make the lookup cheap, block-aligned hashing makes it safe, and the hit rate is what decides whether it is a 2x or a 10x win. The interview question is how you would route to make it hit.

Explore the full curriculum →

DATABRICKS LLM INFERENCE & SERVING FAQ

What LLM Inference & Serving questions does Databricks ask in interviews?▲

Databricks's ML Platform Engineer loop draws llm inference & serving questions such as "How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?", "Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?", "What is the KV cache, and why does it keep growing while a request is being served?". Prefill versus decode, the KV cache, PagedAttention and continuous batching, chunked prefill, speculative decoding, disaggregated serving, quantization, vLLM, SGLang and TensorRT-LLM, multi-LoRA and routing: hosting open-weight models at a latency SLO and a cost you can defend. The full set, ordered easy to hard with expert answers, is below.

How should I prepare for the Databricks LLM Inference & Serving round?▼

Does Databricks hire AI infrastructure engineers?▼

What does the Databricks AI infrastructure interview test?▼

Other Databricks interview rounds

The other tracks Databricks's ML Platform Engineer loop tests.

Databricks ai systems design→Databricks distributed training→Databricks cluster orchestration→

Prep the whole Databricks ML Platform Engineer loop

LLM Inference & Serving is one round. Unlock every answer across Databricks's full loop, plus the concept curriculum, for 6 months. One payment, no auto-renewal. Free questions in every track to start.

Unlock all answers · ₹2,000 / $25 Full Databricks interview guide

Independent and not affiliated with Databricks. All trademarks belong to their owners.

Databricks LLM Inference & Serving interview questions

LLM Inference & Serving questions tagged to Databricks

How would you serve hundreds of LoRA adapters on one base model, and what does it cost in throughput?

More LLM Inference & Serving questions for Databricks's loop

Why do prefill and decode behave so differently, and why does that matter for the hardware you serve on?

What is the KV cache, and why does it keep growing while a request is being served?

What is the difference between static and continuous batching, and why did it change LLM serving?

How does PagedAttention work, and what problem was it solving?

Define TTFT, TPOT and goodput, and tell me how you would measure each one in production.

You have eight H100s and Llama 3.1 70B. How many concurrent users can you serve, and what changes the number?

A long prompt arrives while sixty users are mid-generation. What happens, and how does chunked prefill fix it?

When does speculative decoding speed up serving, and when does it break even or hurt?

Compare the KV cache footprint of multi-head, grouped-query and multi-head latent attention with numbers.

You need to quantize a model for serving. Which method do you pick, and what do you measure before shipping it?

When does splitting prefill and decode onto separate GPU pools pay for itself, and what does the KV transfer cost?

Every request shares a 2,000-token system prompt. How does prefix caching exploit that, and how does the radix tree work?

You have one GPU and a synchronous API that receives 100 documents at once. Design the batching, and show the latency math.

Your p99 TTFT tripled last night and p50 did not move. Walk me through how you find the cause.

vLLM, SGLang or TensorRT-LLM: which engine do you pick for a new deployment, and what would change your mind?

Concepts behind Databricks's LLM Inference & Serving round

Other Databricks interview rounds

Prep the whole Databricks ML Platform Engineer loop