AI Infra Interviews logo
← Open-Weights Models & Serving Engines / 27
hardTogether AIFireworks AIBaseten

A model interleaves linear-attention and full-attention layers. What changes about serving it?

Budget growing attention state, bounded recurrent state and retained prefix snapshots separately. Work the short-context crossover, then test whether the cache manager resumes every layer from the same prefix boundary.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Budget growing attention state, bounded recurrent state and retained prefix snapshots separately. Work the short-context crossover, then test whether the cache manager resumes every layer from the same prefix boundary.

The full answer is part of Premium. A free account includes more answers in each topic, but does not unlock this one.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 316 remaining answers · ₹2,000 / $25

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Foundational
🧮 Open Weights & Serving Engines
Reading config.json to Size a Model You Have Never RunRead a model configuration as an architecture description, then combine it with tensor metadata and engine allocation rules. Learn which fields determine attention state, why mixed precision needs tensor-level accounting, and how to turn a first estimate into a testable per-GPU capacity plan.
Advanced
🚀 Inference & Serving🔒 Premium
Hybrid State Checkpoints and RollbackA hybrid model must restore attention history and recurrent state to the same token boundary. Learn why a cache hit can require replay, how rejected draft tokens corrupt an in-place state, and what saved checkpoints cost.
Foundational
🧮 Napkin Math & Capacity
KV Cache SizingDerive cache payload from the stored attention state, then account for mixed sequence lengths, block allocation and per-device limits. Worked examples distinguish exact bytes from deployable capacity.
Foundational
🧮 Open Weights & Serving Engines
Capacity Planning for Open-Weights FleetsMatch forecast traffic to a measured replica operating envelope, then check failure domains, rollout capacity, readiness time and cost. Output tokens alone omit prompt work and live state; total parameter count alone does not determine a replica's throughput.
Explore the full curriculum →
UP NEXT ON YOUR JOURNEY
Next in this trackYou serve a small, a medium and a large open-weights model. How do you route requests between them?Next in this trackYou need a released bf16 model at half the footprint. Walk through quantizing it yourself.Next in this trackDesign a deployment that serves a trillion-parameter model at a million tokens of context with usable latency.
FEDITOR'S NOTE

A high KV hit rate can coexist with expensive recurrent-state recomputation. Ask which cache group produced the hit and how many prompt tokens the engine actually skipped.

DISCUSSION · 0

No comments yet — be the first to share your approach.