data parallel
AI infra interview questions tagged data parallel, across every topic.
3 questions · 1 unlocked for you
Concepts behind "data parallel"
The curriculum that explains the ideas these questions test.
Foundational
Multi-Node Serving TopologiesOnce a model needs more GPUs than one NVLink domain holds, the deployment shape becomes a real design decision. Tensor parallelism stays inside the node because it communicates twice per layer per token. Across nodes the choices are data parallelism with replicas, pipeline parallelism with a bubble, expert parallelism with an all-to-all, or disaggregation that runs prefill and decode on separate pools and ships the KV cache between them. Each has a different failure mode and a different scaling story.🧮 Open Weights & Serving Engines
Advanced
Topology-Aware CommunicationThe same collective can run at 900 GB/s or at 50 GB/s depending on which links it is laid across, so the mapping of parallel groups onto hardware is a performance decision, not a deployment detail. The rule: tensor-parallel groups inside the NVLink domain, data-parallel rings along rails, pipeline stages across the fabric, and every rank placed so its partner is one hop away. NCCL discovers the topology and does most of this when the job lets it; the failures come from placements that do not.🔌 Networking & Storage🔒 Premium