What is continuous batching?

Continuous batching lets an inference server add and remove requests from a running batch at every generation step, instead of waiting for a whole batch to finish together. New requests start immediately, finished ones free their slot, and GPU utilization stays high under mixed traffic.

Static batching forced a tradeoff: big batches for throughput, small ones for latency, and every request waited for the slowest in its group. Step-level scheduling dissolves the tradeoff, which is why every serious serving stack now uses it.

For buyers this is invisible but priced in: platforms that batch well serve the same model for less, and that difference flows into per-token list prices.

Related terms