Advanced

Tensor Parallelism

Tensor parallelism splits operations within a layer across GPUs. Derive the column/row matrix split, count training and inference collectives, and choose a group size from memory, latency and actual interconnect performance.

This concept is part of Premium, which includes all concepts, course lessons and answers.

Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew

RELATED CONCEPTS

Pipeline Parallelism and the Bubble ZeRO and FSDP Collective Communication Primitives NVLink, NVSwitch and PCIe Topology-Aware Communication Attention Variants: MHA, GQA, MQA and MLA

LESSONS THAT TEACH THIS

Training at ScalePrice parallelism by message size, frequency and exposed time26 min→Inference EngineeringSplitting a model across devices buys memory and spends interconnect15 min→Training at ScaleMap communication groups onto measured physical paths24 min→

PRACTICE THIS IN REAL QUESTIONS

Distributed Training & ParallelismDoes tensor parallelism have to stop at 8? Show me the numbers.→Napkin Math, Cost & CapacityEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s→Distributed Training & ParallelismCompare data, tensor and pipeline parallelism. What does each one shard, what does each one communicate, and where does each one live?→Napkin Math, Cost & CapacityHow long does an all-reduce of a 70B model's gradients take on eight GPUs?→Distributed Training & ParallelismHow do sequence parallelism and context parallelism make 128k-context training of a 405B possible, and what do they cost?→Distributed Training & ParallelismYou need to train a 100B dense model and it does not fit on one node. Walk me through how you would lay it out.→