communication

Count gradient traffic from a named dtype, separate initialization and buffer broadcasts, and measure the reduction tail instead of assuming it is hidden.

Count tensor-parallel messages and compare their payload service time on two explicit paths. Explain why TP8 is a common placement choice, not a fixed ceiling.

Derive the weighted gradient, test a short micro-batch, and separate activation savings from total memory and communication. Matching the optimizer step is only part of matching a larger batch.

Overlap starts with a dependency: communication must finish before its consumer needs the result. Estimate the available window, then measure readiness, contention and complete step time for DDP, FSDP or tensor parallelism.

Tell an incident through the decisions you made, the evidence available at the time and the cost of recovery. Keep your role clear and distinguish mitigation from finding the root cause.

Coordinate simultaneous incidents with the people available. Contain urgent harm, assign next actions and revisit priorities as evidence changes.

Report a capacity shortfall with assumptions, consequences, feasible options and a decision deadline. Show how one uncertain project changes the forecast.