AllocateWhat is the KV cache?
The KV cache stores the attention keys and values for every token a model has already read, so each new token only attends over cached state instead of reprocessing the whole prompt. It is what makes generation fast, and it is the memory cost that grows with context length.
The cache is why long context is expensive: a 70B-class model spends roughly 0.33 MB of memory per token of context at FP16, so 100,000 tokens of context holds about 33 GB beyond the weights themselves.
Serving stacks fight for this memory with paged attention, cache quantization, and prompt caching, because every gigabyte of KV cache is a gigabyte not available for batching more users onto the same GPU.
Try it: GPU VRAM calculatorRelated terms
Allocate is the cloud inference platform for companies that want to train and run their own models.