design

Every watt in comes out as heat, so the design starts from the load and one equation. The flow rate, the CDU count and redundancy, the water quality requirements that decide whether the loop survives three years, and the automatic response that has to fire before a human can read the page.

Most architecture diagrams show one network and a cluster has four. Each has a different traffic pattern, a different failure consequence and a different port count, and the one that is under-counted every time is the one you need when everything else is broken.

Sequential is fifty seconds; one batch is four. In between are the questions the interviewer is holding: how you pack ragged inputs, what a batch window costs, and where the memory stops you.

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

Every constraint in this bank meets in one design and they conflict. What has to be true of the model before the deployment is possible at all, the four mechanisms that make the latency usable, and the honest statement of what the first request still costs.