TL;DR: Placement changes the paths used by the job's communication groups. Map ranks, GPUs and NICs, then compare matched traces before blaming topology. A compact placement can improve throughput, but the scheduling decision should minimize completion time or meet the stated SLO, including time spent waiting for that placement.
How to approach it
First check that both runs use the same model, inputs, parallelism and software. Identify which group crosses which links. Compare exposed communication and useful update duration, then evaluate waiting versus starting on a less favorable layout.
A strong answer
Two eight-GPU jobs have different throughput. One is on a single node; the other uses multiple worker pods across two nodes. That is a plausible topology difference, not a complete diagnosis. Eight GPUs could be data-parallel replicas, a tensor-parallel group or another layout. A single Kubernetes pod cannot be split across nodes, so first establish the workload's actual structure.
Topology-Aware Scheduling connects physical locality to the allocation. Record rank-to-device UUID, node, GPU interconnect domain, NIC and NUMA mapping (which CPU socket's memory and PCIe lanes each device sits closest to). Do this after the framework has created its groups; launcher inputs alone may not show the final assignment. Check whether the transport actually uses the intended interface.
Tensor parallelism often has frequent communication, but its cost depends on message sizes, implementation and the overlap schedule. Data parallelism, expert routing and pipeline traffic have different patterns. Even inside one node, PCIe roots and GPU interconnects can differ. Do not assume that all local paths are equal or that cross-node execution is necessarily incorrect.
NCCL's environment documentation provides configuration and diagnostic controls, but forcing an interface or algorithm without a baseline can make another workload worse. Capture the selected path and compare one change at a time.
Use a controlled experiment: the same workload, seeds where relevant, shapes, clock/power settings, warm-up and background load on each placement. Collect useful tokens or updates per second, rank arrival times, collective durations, device activity and network counters. If one rank reaches a collective late because of data loading, the collective's apparent duration can look like a network problem.
A synthetic decomposition illustrates the need to measure exposed time:
Placement A: 800 ms compute + 100 ms exposed communication = 900 ms per step
Placement B: 800 ms compute + 300 ms exposed communication = 1,100 ms per step
B duration / A duration = 1,100/900 ≈ 1.222
B throughput / A throughput = 900/1,100 ≈ 0.818
Sanity: 3× exposed communication does not imply a 3× slower whole step.
Those values are illustrative, not a predicted NVLink-to-network ratio. Nominal link bandwidths may use different directional conventions and cannot be converted directly into application speedups.
Now include queue time. Suppose a job takes ten minutes on a compact layout and fourteen on an available alternative. Waiting three minutes for compact capacity finishes in thirteen, one minute earlier. The break-even wait is four minutes. For an uncertain queue, compare a distribution and deadline risk rather than a single optimistic estimate. Long training runs can justify a larger wait; short interactive jobs may not.
Use hard locality requirements when the job cannot meet memory, correctness or required service constraints otherwise. Use preferences when slower placement remains acceptable. Reverse a strict-locality policy when fragmentation or queue delay outweighs the measured runtime benefit. Reverse the topology diagnosis when matched placement changes do not improve the job or when a late producer explains the difference.
What interviewers probe next
- "Why log device UUIDs rather than ordinals?" Visibility and enumeration can change local numbering.
- "Can a group span nodes successfully?" Yes, if the implementation and performance requirements support it.
- "What should a placement score optimize?" The declared objective, such as completion time, with hard constraints enforced first.
Common mistakes
- Inferring tensor parallelism from GPU count alone.
- Dividing nominal link bandwidths to predict step speedup.
- Ignoring queue delay when enforcing locality.
Key takeaways
- 900 ms versus 1,100 ms is about an 18.2% throughput reduction.
- A ten-minute compact run beats a fourteen-minute alternative only if its wait is under four minutes.
- Verify group membership and actual transport together.