BLOG ILLUSTRATION / EXECUTION STORY

Reschedule unfinished requests

The flow shows text, token IDs, scheduled GPU execution, kept K/V, sampling and the next scheduler step. Model forward and GPU work are views of the same execution.

Reschedule unfinished requestsA scheduler selects request work. GPU kernels implement model forward. This computation reads previous K/V and appends new K/V. Logits supply a sampled token, and unfinished requests return to scheduling. Model and GPU views show one execution. Request text text + metadata Token IDs formatted sequence Scheduler select the work Forward pass GPU execution New token sample + stream KV cache kept keys + values unfinished → reschedule read append
The scheduler selects work. GPU kernels implement model forward, and K/V stays in memory. Unfinished requests return to scheduling. Model and GPU boxes show the same execution, not two serial runs. The author made this diagram. The playback is an example, and animation time is not measured runtime. Open full-size SVG