← Napkin Math & Capacity
Advanced
Communication Volume Estimates
Estimate communication by naming the tensor, collective, group and synchronization schedule. Work through per-rank ring bytes, conditional sharding costs, activation exchanges and routing imbalance before converting traffic into exposed time.
This concept is part of Premium, which includes all concepts, course lessons and answers.
Get Premium · ₹2,000 / $256 months of access · one payment · no auto-renew
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
Networking, Interconnects & StorageName the four collectives a training job uses, say what each moves, and match them to the parallelism that needs them.→Networking, Interconnects & StorageDoes multi-node inference need the same fabric as training, or can it run on ordinary networking?→Networking, Interconnects & StorageMap a 1,024-GPU job's parallelism onto the hardware. Which dimension goes on NVLink, which on the fabric, and what does a wrong order cost?→Napkin Math, Cost & CapacityEstimate the latency of one decode step for a 70B model under tensor parallelism across eight H100s→Napkin Math, Cost & CapacityHow long does an all-reduce of a 70B model's gradients take on eight GPUs?→GPU & Accelerator ArchitectureNVLink versus PCIe: when does the interconnect actually matter for an LLM workload, and when can you get away with PCIe?→