AllocateWhy does GPU memory bandwidth matter for inference?
Generating a token requires streaming every active model weight from GPU memory through the compute units, once per token. For single-stream generation the accelerator is usually waiting on memory, not math, so memory bandwidth, not raw compute, sets the ceiling on decode speed.
This is why quantization speeds up inference as well as shrinking it: 4-bit weights move a quarter of the bytes of FP16, so the same bandwidth streams the model in a quarter of the time.
It is also why batching is the economics of serving: the weights stream once per step regardless of batch size, so serving 30 users costs little more bandwidth than serving one. Prompt processing, by contrast, is compute-bound, which is why input and output tokens are priced differently.
Try it: GPU VRAM calculatorRelated terms
Allocate is the cloud inference platform for companies that want to train and run their own models.