AI Infra Interviews logo

Define when requests join, leave and wait

A batch-size limit does not decide when requests start or stop. Continuous batching can refill finished positions between iterations, but admission must also account for cache growth, waiting time and overload.

13 MIN · PLUS

This lesson is included with a free account. Sign in to continue reading.

free account · no card needed