mediumNewAnyscaleFireworksDatabricks

Design serving for 100 fine-tuned variants of one 70B base. Multi-LoRA on shared replicas or a replica per variant?

A replica per variant is 100 copies of a 140 GB base and 800 GPUs mostly idle; multi-LoRA holds the base once and hundreds of adapters in a few gigabytes. The memory arithmetic, the throughput cost of mixed batches, the routing that keeps hot adapters resident, and the two cases where a variant earns its own replica.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🚀 Inference & Serving🔒 Premium

Multi-LoRA ServingA LoRA adapter is a few hundred megabytes of low-rank matrices that turn a base model into a fine-tuned variant, and multi-LoRA serving runs hundreds of them on one copy of the base weights by keeping the adapters in memory and applying the right one per request inside the batch. It is how a platform serves a thousand customers' fine-tunes without a thousand deployments. The costs are an extra small matmul per layer, adapter memory and loading, and a scheduler that has to batch across adapters without starving any of them.

Foundational

📐 AI Systems Design

Multi-Region Serving and FailoverRunning inference in more than one region buys latency for distant users and survival when a region fails, and it costs a second fleet that must be capable of absorbing the first one's traffic. The design turns on three decisions: whether regions are active-active or active-passive, what state has to cross regions and what deliberately does not, and how much headroom each region carries so a failover does not simply move the outage.

Foundational

🚀 Inference & Serving

The KV CacheThe KV cache stores each token's attention keys and values so decode never recomputes them, turning a quadratic cost into a linear one at the price of memory that grows with every token in every concurrent sequence. Its size, 128 KB per token for Llama 3.1 8B and 320 KB for 70B in bf16, is what caps concurrency and context on a given GPU, so it decides batch size, replica count and whether a model fits at all.

Advanced

📐 AI Systems Design🔒 Premium

Multi-Tenant Fine-Tuning ServiceA fine-tuning service takes a customer's dataset and a base model and returns a model, and the design problem is that many customers want this at once, cheaply, without seeing each other's data, on GPUs that must not sit idle between jobs. LoRA changes the shape: an adapter is a few hundred megabytes rather than a copy of the base, so many jobs can share a base in memory and many adapters can be served from one replica. This page designs the service end to end: the pipeline, the LoRA arithmetic that sets memory and cost, the isolation, the scheduler that packs jobs, and the serving path.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackDesign autoscaling for an LLM service whose traffic triples within a minute. What signal, what cold-start budget, what warm pool?Next in this trackDesign a KV cache tier across GPU memory, host memory and NVMe so prefixes survive across sessions. Bandwidths, and when it pays.Next in this trackDesign the registry and deploy pipeline for model weights: 141 GB artifacts, versioning, distribution to a thousand nodes, integrity.

FEDITOR'S NOTE

Scored on the memory arithmetic (base once, adapters at a few hundred megabytes), on the batching cost of mixed adapters and how it is bounded, on adapter placement and routing, and on knowing when a full fine-tune or a hot variant breaks the shared design.

DISCUSSION · 0

No comments yet — be the first to share your approach.