AllocateWhat does tokens per second measure?
Tokens per second measures generation speed: how fast a model streams output once it starts. Per-user decode speed decides how a response feels; aggregate throughput per GPU decides what serving costs. The two trade off against each other through batch size, and providers tune that balance.
Model size dominates the number. As planning figures, a batched H100-class GPU sustains on the order of 2,400 output tokens per second across users on an 8B model, but nearer 260 on a 300B-class one; those are the conservative ladder steps our break-even calculator uses.
Quantization, speculative decoding, and batching all move the number, which is why the same open model runs at different speeds on different platforms.
Try it: Self-hosted vs API break-evenRelated terms
Allocate is the cloud inference platform for companies that want to train and run their own models.