BLOG ILLUSTRATION / EXECUTION STORY

One chat request across engine processes

Two source-derived traces follow one streaming chat request on one GPU. vLLM detokenizes in the API-server process. SGLang sends token IDs to a separate detokenizer process. Select either trace on this page.

One chat request across engine processes A frontend process sends token IDs to a scheduler inside the engine process. The scheduler launches a batch on a GPU worker in the same process and receives sampled tokens. New tokens return either directly to the frontend or through a separate detokenizer process, which sends text to the frontend. Engine process Frontend HTTP + tokenizer Scheduler batch + KV GPU worker model runner Detokenizer own process token IDs new tokens batch tokens token IDs text Dashed box: one OS process
Author-created trace from vLLM v0.32.0 and SGLang v0.5.23 with default settings and one GPU. With tensor parallelism above 1, both engines add worker or scheduler processes per GPU. Playback time is not measured latency. Open full-size SVG