Networking, Interconnect & Storage Interview Questions

Separate the all-reduce result from NCCL execution choices, and compare latency models without treating them as universal selection rules.

Track actual values across ranks, then count bytes with an explicit buffer definition. Reduction combines values; gathering and all-to-all change ownership.

Choose between supported InfiniBand and RoCE designs using workload tests, congestion behavior and a complete cost model.

Trace registered-memory access and GPU staging, then compare RDMA with sockets using measured CPU and application performance.

Follow corresponding GPU positions through a rail, including the leaf and spine links needed when one rail outgrows a switch.

A bandwidth regression narrows the search only after the experiment is defined. Work through rank-count normalization, controlled path tests and the difference between collective time and training progress.

Count endpoint, leaf, aggregation and core ports for an explicit 4,096-GPU design. Compare 1:1 and 2:1 without confusing switch ports, links or optics.

Calculate queue growth during incast, then distinguish endpoint rate feedback from link-level priority pause and shared-resource contention.

Compare serialized and overlapped storage paths, then test whether direct GPU I/O improves the actual pipeline stage.

Design a durable dataset and optional working-set cache around access semantics, request overhead and versioned readiness.

Measure batch readiness and producer capacity, then size prefetch for bursts without hiding a sustained throughput deficit.

Plan a reproducible tokenization build: measure CPU demand, choose a safe token dtype, publish complete shards and distinguish token lookup from training resume.

Design snapshot ownership, background writes and manifest publication separately, with explicit checkpoint bytes and recovery scope.

Construct concrete TP and DP groups, then verify physical devices and estimate traffic with per-rank rather than whole-model bytes.

Separate NCCL logging, device selection and execution overrides from PyTorch failure handling. Work through a faster collective that makes the job slower, and learn which timeout actually fired.

Separate host-buffer, GPU-buffer and collective tests, with explicit message-size models and bandwidth metric definitions.

Derive bandwidth from actual per-rank traffic and readiness deadlines, showing why bucketing cannot erase a capacity shortfall.

Explain packet spraying through a small path-collision model, then separate transport features from measured performance and a supported fabric purchase.

Derive a complete rail topology and distinguish physical links, switch ports and transceivers. Size storage and management from their own requirements.

Compare tensor, pipeline and expert traffic with per-request KV transfer. Work through declared payload and latency assumptions before choosing a serving network.

Trace a link flap through transport and job recovery. Calculate detection, restart and replay costs, and learn what continuous counters and timed tests can actually establish.

Calculate how many reads a training loader needs in flight, then see when larger requests waste bandwidth. Worked examples separate useful samples, fetched bytes, prefetch and the limits of Little's Law.

Budget exact hashes, MinHash signatures and candidate-pair growth separately. Use an executable probability model and measured filter rates to choose stage order.

Follow expert assignments from logical payload to off-rank and cross-node traffic. Compare dispatch and combine dtypes, receiver skew and local-domain placement.

Can one storage system serve datasets and checkpoints? Follow a save from staging to recovery, calculate its budget beside protected reads, and learn when separate paths actually solve the contention.

Follow pause requests against the direction of data, estimate queue growth and distinguish cascade evidence from its possible causes.

Derive the two-tier endpoint limit, audit explicit 8,192-endpoint constructions and compare switches, links, optical modules and failure paths.

Distinguish GPU-memory capability, registration and selected transport. Compare direct and staged paths without relying on a half-speed rule or one log substring.

Two tenants can fit the average storage budget and still miss both checkpoint deadlines. Work through reservations, borrowing and bounded admission while protecting training reads, then test the design under failure.

Investigate a firmware-correlated regression with matched metrics, mapped paths and a controlled canary. Separate a configuration hypothesis from evidence that confirms it.