What problem does KV Cache solve?

It avoids recomputing past K/V projections during autoregressive generation.

What is the one-line intuition for KV Cache?

It changes where work or state lives so the system can use hardware more efficiently.

Stored K and V tensors per layer, per active request, and per cached token.

\(\mathrm{KV\ bytes}=2\cdot B\cdot L\cdot N_{\mathrm{layers}}\cdot N_{\mathrm{kv\ heads}}\cdot d_{\mathrm{head}}\cdot b\).

It determines how much concurrency and context length fit on each serving GPU.

Memory grows linearly with batch, context length, layers, KV heads, head dimension, and dtype size.

Decode often becomes memory-bandwidth-bound because every new token reads a large cache.

Fragmented cache allocation can waste memory even when total free bytes look sufficient.

Inspect KV memory usage, TPOT, cache hit rate for prefix caching, and eviction or block fragmentation.

Caching saves compute but spends persistent GPU memory.

Requests enter, extend, and finish at different times; block allocation avoids repeatedly moving large contiguous KV buffers.

It stores fewer K/V heads than query heads, so the \(N_{\mathrm{kv\ heads}}\) term in the cache formula decreases.