What is paged attention?

Paged attention manages the KV cache the way an operating system manages RAM: in small fixed-size blocks allocated on demand, instead of one contiguous slab per request. It removes the memory fragmentation that wasted most cache space in earlier servers, so far more concurrent requests fit on one GPU.

Before paged attention, servers reserved worst-case cache space for every request up front; a chat that might reach 32K tokens held 32K tokens of memory from its first word. Block-based allocation ends that, and blocks can be shared between requests with a common prefix.

The technique, popularized by modern open-source serving engines, is a large part of why per-token prices for open models fell: the same hardware serves several times the traffic.

Related terms