TL;DR: Specify which requests reach retrieval, reranking and generation before sizing any tier. Convert each branch to shard queries, candidate pairs and prompt/output tokens. Keep cache identity tied to access and source freshness, and validate the complete latency distribution. Published FLOPs or an assumed cache hit rate cannot establish a production fleet size by themselves.
How to approach it
Clarify corpus size, access rules, update freshness, query rate and which queries need generated answers. Decide whether a cached response includes the result list as well as the answer. Trace both hits and misses, then size every stage using measured capacity on its workload. Close with degraded behavior when retrieval or generation cannot meet its contract.
A strong answer
The pipeline has three tiers: retrieval finds candidate documents in sharded indexes, reranking uses a second model to reorder those candidates by relevance, and generation writes the answer from the top few. Decide what a cache hit returns before drawing the rest of the system. If it contains both search results and the generated answer, the request can skip retrieval. If it contains only the answer and fresh results are still required, retrieval must run. Carry the same choice through the diagram and capacity calculation.
Hypothetical peak and branch fractions:
ingress = 10,000 queries/s
40% receive a valid cached complete response → 4,000 hits/s
retrieval misses = 6,000/s
eight document shards queried per miss → 48,000 shard queries/s
rerank 50 candidates per miss → 300,000 query-document pairs/s
30% of misses require generation → 1,800 LLM requests/s
each generation: 2,500 uncached input tokens and 200 output tokens
input demand = 4.5 million tokens/s; output = 360,000 tokens/s
Sanity: 4,000 cache hits + 6,000 misses = 10,000 ingress queries/s.
Replace these assumed fractions with traffic measurements when sizing a real service. If every request needs fresh search results, retrieval receives all 10,000 queries/s even when some answers are cached. Also test what happens when a model or corpus update invalidates the cache. The Inference Platform Architecture supplies the request boundaries; this design adds access-scoped retrieval and freshness.
An embedding-only payload of 100 million vectors at 64 compressed bytes each is 6.4 GB. IDs, graph/list structures, stored passages, filter metadata, replicas and update buffers add to it; that arithmetic does not establish a 12 GB total index. Benchmark search quality, memory and latency with the actual index and access filters.
For a hypothetical measured shard capacity of 2,000 queries/s at the required latency and recall, each shard receives 6,000 queries/s. Three replicas per shard exactly cover that rate, and four preserve the same capacity after one replica fails: eight shards times four equals 32 replicas. This leaves no extra performance headroom after that failure. Placement must ensure one host failure does not remove several assumed-independent replicas.
Reranking capacity is measured in pairs with their sequence lengths and batch behavior. If a tested replica handles 10,000 qualifying pairs/s, 300,000 pairs/s needs 30 saturated replicas, or ceil(300,000 / (10,000 × 0.8)) = 38 at 80% target occupancy. This is a hypothetical measurement input, not a claim about any specific GPU or cross-encoder. Reducing candidates can save work but changes retrieval quality; evaluate it before rollout.
For generation, assume separate tested pools provide 45,000 uncached input tokens/s per prefill replica and 25,000 output tokens/s per decode replica on this fixed mix:
At 80% target occupancy:
prefill replicas = ceil(4,500,000 / (45,000 × 0.8)) = 125
decode replicas = ceil(360,000 / (25,000 × 0.8)) = 18
total = 143 replica units, with their GPU layouts specified separately
Sanity: 125 × 45,000 × 0.8 = 4.5 M input tokens/s.
That is a disaggregated sizing illustration. It excludes transfer capacity and is not a promised hardware configuration. A shared pool needs a joint benchmark; isolated prefill and decode rates cannot both be consumed independently on the same GPUs. Capacity Planning and Utilization also requires per-rank memory, prompt-length tails, burst reserve and recovery capacity.
Allocate stage budgets, then validate the end-to-end trace. Stage p95s are not additive. Under an illustrative independent identical-shard model, eight shards all finishing within a deadline has probability F(t)^8; a 95% fan-out success at that deadline requires per-shard F(t)=0.95^(1/8)≈99.36%. Correlated slowdowns invalidate that independence assumption. Hedging adds load, so use measured thresholds and a bounded duplicate-work budget.
Reuse an answer only when the query meaning, current access scope, model and prompt versions, and source freshness all permit it. A permission revocation must invalidate or exclude answers that expose the revoked document. Similar queries can still need different answers, especially when they differ in a number, a negation or a permission. Begin with exact, safely scoped reuse and expand only when correctness and privacy tests support it.
Return timely search results when generation is overloaded if the product permits that fallback, clearly marking absent or delayed generated answers. Bound queues and preserve source citations; an unavailable source cannot be silently replaced with invented evidence. Reverse the caching investment when safe reuse is rare or invalidation dominates, and spend effort on shorter relevant context, model choice or retrieval efficiency instead.
What interviewers probe next
- "What if cache hits fall to zero?" Retrieval rises from 6,000 to 10,000/s and generation from 1,800 to 3,000/s under the same branch rule. Recompute every dependent stage.
- "Can a cache entry survive a permission change?" Only if the current access policy still allows its complete content; TTL alone does not enforce revocation.
- "Why not sum p95s?" Stage durations have distributions and dependencies. Measure the complete request and use the stage timings for diagnosis.
Common mistakes
- Counting cache hits differently in the drawing and fleet arithmetic.
- Treating compressed vectors as total index memory.
- Claiming isolated throughput figures as joint shared-pool capacity.
- Reusing answers across access scopes or relying on similarity alone.
Key takeaways
- The toy branch yields 48,000 shard calls, 300,000 rerank pairs and 1,800 generations per second.
- The separate-pool example needs 125 prefill and 18 decode replica units at 80% occupancy.
- Access, freshness and quality are part of cache eligibility and measured capacity.