{
  "title": "Serial and overlapped scheduler loops",
  "caption": "Author-created trace of SGLang v0.5.23 event_loop_normal and event_loop_overlap. vLLM, TensorRT-LLM and LMDeploy use the same one-step-ahead idea with different code. Playback time is not measured latency.",
  "alt": "Flow diagram: an SGLang scheduler picks and launches batch k, the GPU reads inputs from a FutureMap, and the CPU processes the previous batch's results",
  "sources": [
    {
      "title": "Original author diagram",
      "file": "diagram.svg",
      "sha256": "841fa1c5fa9689e91205f0731b67090a68c042bd106138fab8d69c6bb1cf5bd8"
    },
    {
      "title": "SGLang v0.5.23 event_loop_normal",
      "url": "https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/managers/scheduler.py#L2026-L2058"
    },
    {
      "title": "SGLang v0.5.23 event_loop_overlap",
      "url": "https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/managers/scheduler.py#L2061-L2134"
    },
    {
      "title": "SGLang v0.5.23 FutureMap input gather",
      "url": "https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/managers/overlap_utils.py#L113-L148"
    },
    {
      "title": "SGLang v0.5.23 result copy stream",
      "url": "https://github.com/sgl-project/sglang/blob/v0.5.23/python/sglang/srt/managers/scheduler.py#L5018-L5042"
    },
    {
      "title": "Article evidence and version ledger",
      "file": "../../illustrations/SOURCE.md"
    }
  ],
  "limitations": [
    "The trace shows a plain decode batch. Speculative decoding, grammars and pipeline parallelism add other branches.",
    "The FutureMap relay is used in both loops at this tag. The loops differ in when results are processed.",
    "Steps are a teaching order. Playback duration does not measure latency or GPU idle time."
  ]
}