TL;DR: All-reduce specifies a result; NCCL selects an execution strategy for the communicator, topology, operation and message. Algorithms, protocols and channel counts affect performance. A ring/tree model helps explain tradeoffs, but actual selection and timing must be checked on the installed version and workload.
How to approach it
Start with what every rank should receive. Separate initialization from per-operation planning and asynchronous execution. Use a small cost model to motivate alternatives, then explain how to test a choice without turning a diagnostic override into a permanent default.
A strong answer
Four ranks contribute vectors [1, 2], [3, 4], [5, 6] and [7, 8]. A sum all-reduce gives [16, 20] to every rank. That contract does not require a particular ring, tree or transport. NCCL collective documentation defines the operation semantics.
Communicator setup discovers and exchanges information needed to connect ranks and choose paths. Some initialization and allocation can be lazy, so warm-up effects are not confined to exactly the first collective. A topology description, plugin or environment override can also affect the plan. Record the library version and configuration when comparing two clusters.
A channel is a subdivision of communication work using implementation-selected paths and resources. More channels can expose parallelism and also consume resources needed by compute. Protocols such as LL, LL128 and Simple differ in transfer and synchronization handling. Do not assign each a universal message-size interval independent of hardware and release.
For an idealized ring with S bytes per rank, N ranks, effective per-direction bandwidth B and per-round latency α:
sent bytes per rank = 2(N−1)S/N
rounds = 2(N−1)
approximate time = 2(N−1)S/(NB) + 2(N−1)α
N = 8, S = 1 MiB, B = 40 GB/s, α = 5 microseconds (assumed):
payload time = 1.75 × 1,048,576 / 40e9 ≈ 45.9 microseconds
latency term = 14 × 5 = 70 microseconds
total ≈ 115.9 microseconds
That model excludes many implementation details. A balanced binary tree has logarithmic path depth; at eight ranks, reduce then broadcast traverses about six levels. Six assumed five-microsecond rounds gives a 30-microsecond latency term, but its payload costs depend on chunking, pipelining and topology. A tree is not automatically faster just because its round count is smaller. Ring vs Tree All-Reduce develops the tradeoff.
NCCL includes choices beyond this two-algorithm sketch, including hardware-assisted paths on supported systems. Small messages often make startup costs important; larger messages make bandwidth more important. Neither statement justifies "never use ring below size X" across all environments.
A host call may finish after placing the collective on a CUDA stream, while the GPU work is still pending. Inputs must be ready before that work reads them, and later consumers must wait for the result. A stopwatch around the host call alone can therefore measure enqueue overhead rather than completion. Use stream-aware timing and preserve those dependencies when testing overlap. NCCL and Collective Algorithms connects this execution model to training.
Use documented NCCL logging and configuration to inspect initialization, graph and tuning information. INFO output is not a guaranteed exhaustive per-call trace of every selection. Available logging and profiler evidence depend on version and subsystem settings. Avoid claiming a decision was observed unless the relevant evidence was actually captured.
Benchmark the message sizes, rank count and placement the job uses, including warm-up and concurrent compute where relevant. Compare the default with one supported algorithm override as a controlled experiment. Keep an override only if complete-step performance improves consistently and correctness passes; reverse it when a different size distribution or library upgrade changes the result.
What interviewers probe next
- "Can ranks call different collectives?" Mismatched collective order, counts or datatypes can hang or corrupt behavior; the contract must agree.
- "Does a fast microbenchmark prove overlap?" No. Concurrent kernels may contend for resources or reach communication at different times.
- "What is the output size?" All-reduce retains the full reduced buffer on every rank; reduce-scatter has different ownership.
Common mistakes
- Treating all-reduce as synonymous with ring.
- Comparing aggregate hardware bandwidth with a per-rank send-rate model.
- Timing only the host enqueue call.
Key takeaways
- The toy sum is [16, 20] on every rank, independent of algorithm.
- The assumed eight-rank ring model gives about 116 microseconds, not a benchmark.
- Verify the installed library's actual behavior before retaining a tuning override.