tensor parallelism

Show what each parallelism axis owns, then derive the groups and traffic for an eight-rank layout. Placement follows measured communication and memory constraints.

Separate the 100B memory floor from the training deadline, then compare two 1,024-GPU layouts with their actual state ownership and communication schedules.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Construct a 16384-GPU candidate layout from ownership and batch constraints, with explicit sharding recipes and pipeline idle time.

Estimate a TP8 decode step from weights, cache reads and explicitly assumed collective times, then replace the serial worksheet with a measured critical path.