# What is continuous batching?

Continuous batching lets an inference server add and remove requests from a running batch at every generation step, instead of waiting for a whole batch to finish together. New requests start immediately, finished ones free their slot, and GPU utilization stays high under mixed traffic.

Static batching forced a tradeoff: big batches for throughput, small ones for latency, and every request waited for the slowest in its group. Step-level scheduling dissolves the tradeoff, which is why every serious serving stack now uses it.

For buyers this is invisible but priced in: platforms that batch well serve the same model for less, and that difference flows into per-token list prices.

## Related terms

- [GPU utilization](https://allocate.network/glossary/gpu-utilization.md)
- [Paged attention](https://allocate.network/glossary/paged-attention.md)
- [Batch inference](https://allocate.network/glossary/batch-inference.md)

---

[HTML page](https://allocate.network/glossary/continuous-batching) · [Machine-readable catalog](https://allocate.network/catalog.json)