BLOG ILLUSTRATION / EXECUTION STORY

Serial and overlapped scheduler loops

Two source-derived traces follow one SGLang scheduler iteration. The serial loop processes batch k right after it runs. The overlap loop queues batch k and processes batch k−1 while the GPU runs batch k.

Serial and overlapped scheduler loops On the CPU, the scheduler picks a batch, launches the step, puts the result handle in a result queue and processes results. On the GPU, the forward pass writes sampled tokens to a FutureMap and reads the next input IDs from it. Results are copied to the CPU, and a control path starts the next iteration. CPU GPU Pick batch schedule Launch step run_batch Result queue batch handles Process results stops + output GPU forward batch on GPU FutureMap next input IDs launches writes reads copies to CPU next iteration Solid: data or launch Dashed: control
Author-created trace of SGLang v0.5.23 event_loop_normal and event_loop_overlap. vLLM, TensorRT-LLM and LMDeploy use the same one-step-ahead idea with different code. Playback time is not measured latency. Open full-size SVG