BLOG ILLUSTRATION / EXECUTION STORY

Prefill gives logits for the first output token

The timeline shows queueing, prompt prefill, first-token sampling, decode iterations and completion. Prefill supplies first-token logits, and streaming can overlap subsequent decode.

Prefill gives logits for the first output tokenPrefill makes prompt K/V and last-position logits for the first sampled token. Decode processes generated inputs, and sampled output can stream. Completion releases this request’s references. Reusable shared blocks may stay. Phase widths are schematic, and playback time is not measured runtime. Widths: schematic Queue wait none metadata Prefill prefill first token make K/V Decode 1 next input next token read + write Decode 2 next input next token read + write Decode 3 next input next token read + write Complete done last event release refs Work Output State
Prefill’s last prompt position supplies the first sampled token. Decode then uses generated inputs, and output can stream during subsequent steps. The phase widths are schematic. The author made this explanatory diagram. The playback is an example. Animation time is not measured runtime. Open full-size SVG