disaggregation

Prefill and decode fight over the same GPU and each ruins the other's latency. Putting them on separate pools ends the fight, at the price of shipping every request's cache across the network. The break-even is a number you can derive.

Two hundred milliseconds is about four thousand prompt tokens of prefill on a full node, before any queueing. Every design choice is about not spending that budget twice.

The two phases want opposite machines, so running them together makes every setting a compromise. What splitting buys, the KV transfer that couples the pools, and the arithmetic that says whether the coupling is affordable for this model.

Every constraint in this bank meets in one design and they conflict. What has to be true of the model before the deployment is possible at all, the four mechanisms that make the latency usable, and the honest statement of what the first request still costs.

It depends entirely on which dimension you split the model along, and the three choices differ by two orders of magnitude in what they demand. The per-token traffic for each, the arithmetic that puts tensor parallelism on the fast link and pipeline parallelism anywhere, and where disaggregated serving lands.

Prefill is compute-bound and decode is bandwidth-bound, and one GPU running both does each worse. The pool ratio from the traffic's prompt and output lengths, the KV transfer cost per request and the link it needs, the router that pairs a prefill worker with a decode worker, and the load at which it beats a monolith.