BLOG ILLUSTRATION / EXECUTION STORY

Full blocks and the same cache key are necessary for reuse

Request A prefills two full prefix blocks. Each block has four tokens. If the cache key is the same and cached KV is still in memory, B can use those eight tokens again. With these conditions, it can prefill its new two-token tail.

Full blocks and the same cache key are necessary for reuseRequest A prefills two full prefix blocks. Each block has four tokens. If the cache key is the same and cached KV is still in memory, B can use those eight tokens again. With these conditions, it can prefill its new two-token tail. Request A first prefill Cached KV valid + in memory Request B same prefix + tail p1 p2 p3 p4 KV: p1–p4 p1 p2 p3 p4 p5 p6 p7 p8 KV: p5–p8 p5 p6 p7 p8 b1 b2 — — new tail prefill P = 4 · two full blocks reused prefix: 8 tokens
Request B uses two cached full blocks with the same cache key again, then prefills its new tail. P=4 is an example. Prefill can include more work because of implementation details. Playback does not imply cache hits occur without cache-key checks. The author made this explanatory diagram. Open full-size SVG