Does multi-node inference need the same fabric as training, or can it run on ordinary networking?

Compare tensor, pipeline and expert traffic with per-request KV transfer. Work through declared payload and latency assumptions before choosing a serving network.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Compare tensor, pipeline and expert traffic with per-request KV transfer. Work through declared payload and latency assumptions before choosing a serving network.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🚀 Inference & Serving🔒 Premium

Disaggregated Prefill and DecodeSeparate prefill and decode to control phase interference, then budget the KV handoff. A short isolated transfer can still overload a shared path; cancellation stops generation before it necessarily releases transfer buffers.

Foundational

🧮 Open Weights & Serving Engines

Multi-Node Serving TopologiesChoose what to distribute: complete replicas, tensor operations, layers, experts or request phases. Compare rank-local fit, communication dependencies, workload concurrency and failure domains. A node boundary is not a universal performance boundary, and multiple layouts can be combined.

Advanced

🧮 Napkin Math & Capacity🔒 Premium

Communication Volume EstimatesEstimate communication by naming the tensor, collective, group and synchronization schedule. Work through per-rank ring bytes, conditional sharding costs, activation exchanges and routing imbalance before converting traffic into exposed time.

Foundational

🔌 Networking & Storage

Dataset Lifecycle: Ingest, Shard and RetainFollow a dataset from source records to verified shards, a pinned version and safe retention. Calculate token storage, distinguish caches from authoritative data, and learn why deleting a version does not necessarily free its shards.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackOne fabric link goes down and comes back once an hour. What does that do to a 1,024-GPU training job, and how would you find it?Next in this trackWhy does object storage behave badly for random reads, and what does a training pipeline do about it?Next in this trackDeduplicate and quality-filter a multi-petabyte web corpus. What does that pipeline cost and where does it bottleneck?

FEDITOR'S NOTE

Count the tensors and dependent calls for each placement. Keep hypothetical timing separate from hardware claims and evaluate both first-token and decode latency.

DISCUSSION · 0

No comments yet — be the first to share your approach.