How do sequence parallelism and context parallelism make 128k-context training of a 405B possible, and what do they cost?

At 128k tokens one layer's activations are 73 GB per sequence and attention grows with the square of the length. The arithmetic that forces the sequence onto sixteen GPUs, what a ring of KV chunks costs per layer, why the communication hides, and where Ulysses and Megatron sequence parallelism fit around it.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🕸️ Distributed Training🔒 Premium

Context and Sequence ParallelismContext parallelism splits a single long sequence across GPUs so that attention over 128k tokens fits in memory that would otherwise need terabytes of activations per layer. Ring attention rotates key-value blocks around the ranks while queries stay put, and grouped-query attention is what makes that rotation cheap enough to hide behind the attention math.

Advanced

🕸️ Distributed Training🔒 Premium

Tensor ParallelismTensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Foundational

🧮 Open Weights & Serving Engines

Multi-Head Latent Attention and Sparse IndexersGrouped-query attention shrank the KV cache by sharing key and value heads. Latent attention goes further by caching a single compressed vector per token per layer and reconstructing the heads on the fly, which cuts the cache by tens of times rather than by a small factor. On top of that, sparse indexers pick a few thousand relevant positions per query instead of attending to all of them, turning the quadratic term linear at long context. Both are now standard in open-weights models, and both change how a serving deployment is sized.

Foundational

🕸️ Distributed Training

Data Parallelism and DDPData parallelism gives every GPU a full copy of the model, feeds each a different slice of the batch, and averages the gradients with an all-reduce so every replica takes the same optimizer step. It is the first parallelism every training job uses, and the tokens-per-GPU arithmetic behind it decides whether the communication hides behind the backward pass or dominates the step.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackExplain expert parallelism for an MoE model: what the all-to-all is, how much it moves, and why load balance is the whole problem.Next in this trackYour MoE router sends 90% of tokens to 10% of the experts. What happens to the step, and how do you fix it without hurting the model?Next in this trackYou have 16,384 H100s and a 405B dense model. Choose the parallelism layout and justify the order of the axes.

FEDITOR'S NOTE

Scored on separating the two different things called sequence parallelism, on doing the activation and attention arithmetic at 128k, and on showing why the ring's communication is hidden by the s² compute.

DISCUSSION · 0

No comments yet — be the first to share your approach.