Advanced

Tensor Parallelism

Tensor parallelism splits individual weight matrices across GPUs so each rank computes a slice of every layer, which is how a model whose single layer does not fit one GPU gets trained at all. It costs four all-reduces per transformer block on the critical path, which is why it stays inside the NVLink domain and rarely exceeds 8 ranks.

Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew

RELATED CONCEPTS

Pipeline Parallelism and the Bubble ZeRO and FSDP Collective Communication Primitives NVLink, NVSwitch and PCIe Topology-Aware Communication Attention Variants: MHA, GQA, MQA and MLA

LESSONS THAT TEACH THIS

Training at ScalePrice a parallel dimension by the collective it buys, not by its name15 min→Inference EngineeringSplitting a model across devices buys memory and spends interconnect15 min→Training at ScaleMap the mesh so the chattiest dimension takes the shortest path14 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismWhy does tensor parallelism stop at 8? Show me the numbers.→Napkin Math, Cost & CapacityEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s→Distributed Training & ParallelismCompare data, tensor and pipeline parallelism. What does each one shard, what does each one communicate, and where does each one live?→Napkin Math, Cost & CapacityHow long does an all-reduce of a 70B model's gradients take on eight GPUs?→Distributed Training & ParallelismHow do sequence parallelism and context parallelism make 128k-context training of a 405B possible, and what do they cost?→Distributed Training & ParallelismYou need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.→