AllocateWhat is batch inference?
Batch inference processes large sets of requests without a latency requirement: classify a backlog, embed an archive, score every record in a table. Because the work can be scheduled into idle capacity and batched aggressively, it is the cheapest shape of inference per token.
The dividing line is whether a human is waiting. Interactive traffic buys low latency; batch traffic buys throughput. Mixing them on one route means paying interactive prices for work that never needed them.
A worked example: classifying 5 million documents at 1,000 input and 20 output tokens each is about 5 billion input tokens, which is $250 at a $0.05-per-million model like GPT-OSS 20B and $25,000 at a $5 frontier flagship. Batch workloads are where model choice moves real money.
Try it: LLM price comparisonRelated terms
Allocate is the cloud inference platform for companies that want to train and run their own models.