How do sequence parallelism and context parallelism make 128k-context training of a 405B possible, and what do they cost?

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

Updated Oct 2026 · Learn how AI infrastructure works through explanations, worked examples and diagrams, then practise applying it to interview questions.

Trace a causal query across local and remote KV blocks. Separate sequence and context sharding, count one concrete tensor, and avoid dividing memory twice.

This answer is included with a free account. Sign in to read it and save your progress.

free account · no card needed

Work through the numbers

Change the assumptions in a free calculator, then explain what the result establishes and what still needs measurement.

GPUs and time-to-train calculator

The concepts behind this question

Use these concept explanations to review the ideas behind this question.

Advanced

🕸️ Distributed Training🔒 Premium

Context and Sequence ParallelismContext parallelism distributes a long sequence across devices. Attention then exchanges the information each rank needs instead of keeping the whole sequence locally.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersLatent attention changes what is stored; sparse indexing changes which positions receive expensive attention; recurrent layers summarize history in state. Learn to budget each mechanism separately, including indexer work, cache replication and hybrid prefix snapshots.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPReplicas combine gradients from different data. Follow a two-worker update, fix unequal-token weighting, and see why queued gradient buckets can leave more than one reduction exposed.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.Next in this trackYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?Next in this trackYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.

FEDITOR'S NOTE

Distinguish sequence and context ownership, trace causal attention across remote blocks and count named tensors once. Explain the communication assumptions before claiming a memory fit.

DISCUSSION · 0

No comments yet — be the first to share your approach.