BLOG ILLUSTRATION / EXECUTION STORY

A batch changes with new requests and completion

The example has five scheduler iterations for A, B and C. The layout does not change. The state display records membership and completion.

Continuous batching changes the next batchMembership changes in these five example GPU batches. Iteration 1 runs A prefill, and iteration 2 runs A decode with B prefill. Iteration 3 runs A and B decode, and B is completed after this batch. Iteration 4 runs A decode with C prefill, and A is completed after this batch. Iteration 5 runs C decode, and C is completed after this batch. Mixed prefill/decode scheduling is an assumption, and equal spacing does not show duration.IterationGPU batchmembers + workAfter batch1A entersA · P—2B entersA · DB · P—3—A · DB · DEnd of B4C entersA · DC · PEnd of A5—C · DEnd of CP = prefill work D = one decode step
Mixed prefill/decode scheduling is an assumption. Horizontal spacing shows iteration sequence, not elapsed time. These batches are an explanatory example. Open full-size SVG