Which NCCL environment variables are worth knowing, what does each change, and which should you almost never set?

Four of them diagnose, four configure the machine correctly, and several popular ones override tuning that was already right. Which is which, the symptom each addresses, and why a cluster whose job scripts carry a dozen of these has a configuration problem rather than a tuning one.

Updated Sep 2026 · Grounded in real AI infrastructure interview loops and written to a senior-engineer editorial bar, with every number worked and every diagram hand-built.

more free answers with an account · no card

The concepts behind this question

Ranked by how closely each one overlaps this question's topic, so the first card is the thing to read if the answer above moved too fast.

Advanced

🔌 Networking & Storage🔒 Premium

GPUDirect RDMA and GPUDirect StorageBy default a byte leaving a GPU for the network or the disk makes a detour through host memory, crossing PCIe twice and costing a CPU copy. GPUDirect RDMA lets the NIC read and write GPU memory directly, and GPUDirect Storage does the same for NVMe. The win is not raw bandwidth (PCIe is the ceiling either way) but the halving of PCIe traffic and the removal of the host as a bottleneck, which is what makes collectives run at NIC rate and checkpoints run at drive rate. When it is silently off, everything still works, at half speed.

Foundational

🔌 Networking & Storage

Debugging a Slow All-ReduceA training job reports its all-reduce at a third of what the fabric should deliver, every node passed its health check, and nothing is logged. This page is the isolation order that finds the cause in an hour instead of a day: measure the collective in isolation, split the job until the slow pair or rank appears, then check the specific things that make a link, a node or a placement slow. Most cases end at one NIC, one topology mismatch, or GPUDirect silently off.

Foundational

🧮 Open Weights & Serving Engines

vLLM Server Arguments That MatterA vLLM deployment is mostly decided by a dozen flags, and the ones that matter fall into four groups: how the model is split across GPUs, how memory is divided between weights and cache, how requests are batched, and which specialized backends the model needs. Getting the first two wrong produces an engine that will not start or that runs out of memory under load. Getting the third wrong produces an engine that starts, serves, and misses its latency target by a wide margin.

Foundational

🧮 Open Weights & Serving Engines

SGLang Server Arguments That MatterSGLang's tuning model is different from vLLM's in one way that matters: it exposes the scheduler's aggressiveness and the static memory fraction as direct knobs, and its own documentation gives target values for the runtime signals those knobs move. That makes tuning it a measurement loop rather than guesswork. Aim for a queue of a hundred to a couple of thousand requests, token usage above 0.9, and five to eight gigabytes of free GPU memory after startup, then adjust the flags that move each one.

Explore the full curriculum →

UP NEXT ON YOUR JOURNEY

Next in this trackHow do you establish that the link between two GPU nodes is healthy, and what numbers should each test return?Next in this trackWhy do large training clusters provision roughly 400 gigabits per second per GPU rather than more or less?Next in this trackWhat is Ultra Ethernet trying to fix, and does it change how you would choose a fabric today?

FEDITOR'S NOTE

Scored on separating diagnostic from corrective from overriding variables, on knowing the symptom each addresses, and on treating a long list in a job script as evidence of an unfixed problem.

DISCUSSION · 0

No comments yet — be the first to share your approach.