# What is batch inference?

Batch inference processes large sets of requests without a latency requirement: classify a backlog, embed an archive, score every record in a table. Because the work can be scheduled into idle capacity and batched aggressively, it is the cheapest shape of inference per token.

The dividing line is whether a human is waiting. Interactive traffic buys low latency; batch traffic buys throughput. Mixing them on one route means paying interactive prices for work that never needed them.

A worked example: classifying 5 million documents at 1,000 input and 20 output tokens each is about 5 billion input tokens, which is $250 at a $0.05-per-million model like GPT-OSS 20B and $25,000 at a $5 frontier flagship. Batch workloads are where model choice moves real money.

## See also

- [LLM price comparison](https://allocate.network/tools/llm-price-comparison)

## Related terms

- [Inference](https://allocate.network/glossary/inference.md)
- [Model routing](https://allocate.network/glossary/model-routing.md)
- [Tokens](https://allocate.network/glossary/tokens.md)

---

[HTML page](https://allocate.network/glossary/batch-inference) · [Machine-readable catalog](https://allocate.network/catalog.json)