hardNewFireworksTogether AIDeepSeek

What changes when you serve DeepSeek-V3 compared with a dense 70B model?

671 billion parameters that touch 37 billion per token: the memory bill is for the big number and the bandwidth bill is for the small one, until batch and routing collide. The cache is tiny, and that changes the whole fit.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Core

🚀 Inference & ServingSign in

Attention Variants: MHA, GQA, MQA and MLAThe KV cache scales with the number of key-value heads, and the four attention variants differ exactly there: multi-head keeps one KV head per query head, multi-query keeps one for all, grouped-query shares one across a group, and multi-head latent attention caches a compressed latent instead of keys and values at all. For Llama 3.1 70B that is the difference between 2.6 MB and 320 KB per token; for DeepSeek-V3 it is about 70 KB. The variant a model was trained with is a serving decision made before the first GPU was bought.

Foundational

🧮 Napkin Math & Capacity

KV Cache SizingThe KV cache is the memory that decides how many users a serving replica can hold and how long their context can be. Its size per token comes from four numbers in the model's config file (layers, KV heads, head dimension, bytes per element) and one formula; multiplied by context and concurrency it is the number every capacity plan is built on. This page derives it, works it for four models including an MLA one, and shows the two places candidates get it wrong by a factor of eight.

Foundational

🧮 Open Weights & Serving Engines

Expert Parallel and All-to-All BackendsA mixture-of-experts model can be split two ways and the choice changes everything. Tensor parallelism shards each expert across GPUs, which keeps every GPU busy and reads every expert's shard on every token. Expert parallelism gives whole experts to whole GPUs, which reads only the selected experts but requires an all-to-all to route tokens to them and back. The all-to-all is the cost, its backend is a configuration choice matched to the interconnect, and expert load imbalance is what actually limits the result.

Advanced

🕸️ Distributed Training🔒 Premium

Expert Parallelism for MoEA mixture-of-experts layer runs only a few of its experts per token, so the experts can be spread across GPUs and each token shipped to the ranks that hold its chosen experts. That shipping is an all-to-all in each direction, twice per layer per pass, and its cost plus the load imbalance between experts is what expert parallelism is really about.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do you route requests across replicas to maximize prefix-cache hits without unbalancing the fleet?Next in this trackYou stream tokens to clients over HTTP. What can go wrong, and how does the server handle each case?Next in this trackYou need to serve 128k-token contexts. What breaks first, and what do you change?

FEDITOR'S NOTE

Scored on separating memory (total params) from per-step bandwidth (active params), on the MLA cache arithmetic, and on knowing why expert parallelism replaces tensor parallelism at batch.

DISCUSSION · 0

No comments yet — be the first to share your approach.