configuration

Find which startup step runs out of memory, count the allocations on that GPU, and change the setting that controls the shortage.

Derive a prompt-plus-output contract, calculate per-rank state capacity, and measure the concurrency that meets latency and overload targets.

The nodes joined the cluster, but the first request hangs. Trace how far it got, check worker assignments, and reproduce the blocked operation on the same communication path.

Separate NCCL logging, device selection and execution overrides from PyTorch failure handling. Work through a faster collective that makes the job slower, and learn which timeout actually fired.