hardNewCrusoeLambdaNebius

Explain NUMA and CPU affinity on a GPU node. Why can the wrong socket halve host-to-device bandwidth?

A two-socket GPU node has two PCIe trees, two memory pools and one inter-socket link that every wrong-sided transfer must cross. A dataloader on the far socket shares that link with seven other GPUs' traffic, and its copies run at half speed. The topology map, the arithmetic and the pinning.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 283 remaining answers · ₹2,000 / $25

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🧩 GPU & Accelerator Architecture🔒 Premium

NVLink, NVSwitch and PCIeInside a node, GPUs talk over NVLink at 900 GB/s per H100 through an NVSwitch fabric that gives all eight cards full bandwidth to each other; to the host and to anything outside the node they talk over PCIe at 64 GB/s or a 400 Gb/s NIC at 50 GB/s. That fifteen-fold gap is why tensor parallelism stays inside the eight-GPU domain, why NVL72 changes the serving math for MoE, and why the question "how many GPUs share an NVLink domain?" is the first thing to ask about any cluster.

Advanced

🔌 Networking & Storage🔒 Premium

Data Loading Pipelines for TrainingThe dataloader is the only part of a training job that runs on the CPU, the disk and the network at once, and it is the part most often found starving the GPUs. A pipeline that keeps 1,024 accelerators fed has to read sharded files sequentially, decode and tokenize in parallel workers, prefetch several batches ahead, pin memory for the PCIe copy, and do it deterministically enough to resume mid-epoch. The symptom of failure is a GPU at 30% utilization with nothing wrong on the GPU.

Advanced

🔌 Networking & Storage🔒 Premium

GPUDirect RDMA and GPUDirect StorageBy default a byte leaving a GPU for the network or the disk makes a detour through host memory, crossing PCIe twice and costing a CPU copy. GPUDirect RDMA lets the NIC read and write GPU memory directly, and GPUDirect Storage does the same for NVMe. The win is not raw bandwidth (PCIe is the ceiling either way) but the halving of PCIe traffic and the removal of the host as a bottleneck, which is what makes collectives run at NIC rate and checkpoints run at drive rate. When it is silently off, everything still works, at half speed.

Foundational

🖧 Hardware & Cluster Build-Out

SXM, PCIe and Rack-Scale Form FactorsThe same silicon ships in three shapes and the shape decides the deployment. An SXM module is soldered to a baseboard with a full NVLink mesh and needs 700 to 1,400 W of direct power and usually liquid cooling. A PCIe card slots into a standard server, draws through the slot and a cable, and has no NVLink. A rack-scale system like GB300 NVL72 makes the whole rack one NVLink domain and stops being a server at all. Choosing between them fixes your power, cooling, cabling and scheduling story.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackCompare A100, H100 and B200 for a 70B serving fleet. Which gives the most tokens per dollar, and where do fp8 and fp4 change the ranking?Next in this trackIf you could change one thing about GPU architecture for LLMs, what would it be, and what argues against it?Next in this trackOne node in your cluster shows half the expected NVLink bandwidth in nccl-tests. Walk me through isolating it.

FEDITOR'S NOTE

Scored on a correct picture of the two PCIe trees and the inter-socket link, the bandwidth arithmetic that explains the halving, and the specific commands (nvidia-smi topo, numactl, dataloader placement) that fix it.

DISCUSSION · 0

No comments yet — be the first to share your approach.