It is easy to read "vLLM or SGLang?" as a benchmark contest. Read the source code instead, and a different picture appears. In November 2026, the main open-source LLM engines are built on almost the same design. They split the HTTP frontend from the scheduler. They launch the next GPU step before the CPU has read the previous step's tokens. They keep the KV cache in pages and can reuse shared prompt prefixes. Most of them capture CUDA graphs for decode.

The differences that remain are at the boundaries. Where does tokenization run, and how many processes does a request cross? Does the prefix index store hashed blocks or a token tree? When the output is speech instead of text, what is a "stage"? When an RL trainer needs the GPU back, how does the engine release memory and take new weights?

This post compares ten projects at those boundaries. It covers three layers:

• LLM engines: vLLM and SGLang in detail, with TensorRT-LLM and LMDeploy as contrasts.
• Omni serving: vLLM-Omni and SGLang-Omni, plus SGLang Diffusion.
• RL post-training: verl and slime in detail, with OpenRLHF and AReaL as contrasts.

If prefill, decode, continuous batching or PagedAttention are new to you, start with The Life of a Request Inside vLLM. This post assumes those terms.

The LLM engines now share one skeleton

Three changes made the engines look alike. vLLM removed its V0 engine. V1 is now the only engine in the codebase. TensorRT-LLM removed its TensorRT-engine backend in Release 1.2. Its release notes now say that PyTorch is the only execution backend. And both vLLM and SGLang added experimental Rust frontends. vLLM's Rust server keeps the same ZMQ boundary to the Python engine. SGLang's embedded Rust server moves the HTTP, tokenizer and detokenizer roles into Rust threads.

The table compares the four engines with default settings.

TurboMind is LMDeploy's C++ engine.

The skeleton has three parts. A frontend accepts HTTP requests and turns text into token IDs. An engine loop picks the requests for each step and manages KV-cache memory. A GPU worker runs the model. The projects draw the process lines in different places. Fig. 2 follows one streaming chat request through vLLM and SGLang with one GPU.

vLLM: two processes and a ZMQ boundary

In vLLM, the API-server process does the text work. For a chat request, OnlineRenderer.render_chat applies the chat template and tokenizes. AsyncLLM then sends an EngineCoreRequest to the engine. The frontend binds a ZMQ ROUTER socket for inputs and a PULL socket for outputs. Messages are msgpack-encoded.

The EngineCore process runs a busy loop. It reads new requests, schedules a step and runs the model. With one GPU, vLLM uses the uni executor, so the GPU worker lives inside the EngineCore process. With tensor parallelism (TP) above 1, EngineCore broadcasts each step through a shared-memory ring buffer to one worker process per GPU.

The scheduler checks EOS tokens, stop token IDs and max_tokens inside EngineCore. Stop strings are different. Detokenization and stop-string matching run in the frontend OutputProcessor. When a stop string matches, the frontend sends an abort back to EngineCore. This keeps the engine free of text work. The cost is one extra message for each stop-string hit.

SGLang: three roles, one process per TP rank

SGLang uses three roles instead of two. The HTTP server and TokenizerManager share the main process. The Scheduler and DetokenizerManager are subprocesses. All hops are ZMQ PUSH/PULL over ipc:// sockets.

The Scheduler process builds the model worker itself, so a step has no extra process hop to the GPU. SGLang starts one Scheduler process per TP rank. Only TP rank 0 reads from the tokenizer. It broadcasts new requests to the other ranks over a CPU (gloo) group, and each rank runs its own scheduler on the same request list.

SGLang checks stop strings in the Scheduler, on a decoded tail of the output. The detokenizer only trims the matched string. A request can therefore finish without a round trip to the frontend. The cost is a third process and two ZMQ hops on the way out.

Data parallelism repeats the pattern

Both engines scale out by copying the skeleton. With data parallelism (DP) above 1, vLLM starts one EngineCore_DP{i} process per rank. With the internal load balancer, it also starts one API server per rank by default. A DPCoordinator process publishes load numbers and keeps MoE collectives in step. An idle rank runs a dummy forward so that expert-parallel all-to-all operations stay aligned.

SGLang adds a DataParallelController process that sends each request to one DP rank's scheduler. It also has attention data parallelism, now set with --attn-dp-size. Attention runs data-parallel inside one tensor-parallel group, with one scheduler per attention group, while the feed-forward layers stay tensor-parallel. SGLang broadcasts each request only inside its attention group, so each group holds the KV cache for its own requests.

Overlap hides CPU time by running one step ahead

A decode step for a large batch takes a few milliseconds of GPU time. The Python work around it also takes time: pick requests, build inputs, read sampled tokens back, update each request and stream output. If these run one after the other, the GPU is idle during the CPU part of every step.

All four engines now remove that gap the same way. The engine launches the next step before the CPU has read the tokens of the current step. A buffer on the GPU gives the next step its input IDs. Fig. 3 shows SGLang's two loops.

SGLang. Overlap is on by default. In event_loop_overlap, run_batch launches batch k on a forward stream and puts the result handle in result_queue. The loop then calls process_batch_result for batch k−1 while batch k runs. The input IDs for batch k come from the FutureMap, a GPU buffer indexed by request slot. The scheduler gathers output_tokens_buf[req_pool_indices] on the GPU, so the CPU never needs batch k−1's tokens to launch batch k. The result copy to the CPU runs on a separate copy stream.

vLLM. async_scheduling resolves to true by default, and the engine step becomes step_with_batch_queue with two batches in flight. The AsyncScheduler counts tokens that are not yet known as output placeholders. In the default Model Runner V2, a Triton kernel writes the last sampled tokens straight into the next step's input_ids on the GPU.

TensorRT-LLM. The executor picks _executor_loop_overlap unless overlap is disabled or pipeline parallelism is on, and disable_overlap_scheduler defaults to false. Each iteration passes the previous batch's sample state, still on the device, into the next forward. The host then updates the previous batch while the current batch runs.

LMDeploy. The PyTorch engine loop always dispatches step N+1 before it waits for step N. There is no flag. TurboMind, the C++ engine, double-buffers batch data between a scheduler thread and a model thread.

The price is bookkeeping

Running one step ahead has costs, and each engine has code for them.

• Stray tokens. A request that finished one step earlier can already be in the next batch. Its extra token must be dropped and its extra KV slot freed. SGLang skips such requests when it processes results. vLLM skips output for finished requests and caches only committed tokens in the prefix cache.
• Grammars. Structured output needs the previous token before it can mask the next one. vLLM defers only the sampling half of the step. SGLang delays sampling until the previous batch is processed. TensorRT-LLM refreshes its grammar matcher before it samples. None of the three turns overlap off for grammars.
• Speculative decoding. This is the main reason overlap is switched off. vLLM disables async scheduling for draft methods outside EAGLE/MTP, draft models, GPU n-gram, DSpark and DFlash. TensorRT-LLM disables overlap for NGram and some other modes. SGLang runs speculative decoding through its "spec v2" path when overlap is on.

The design lesson is about state ownership. Overlap works only when the state that the next step needs, such as sampled tokens and draft tokens, stays on the GPU. A feature that needs that state on the CPU between steps breaks the overlap.

Prefix caching: hashed blocks or a radix tree

All four engines can reuse the KV cache of a shared prompt prefix. They store the index in different data structures.

vLLM hashes blocks. A block holds 16 tokens by default. Its key is hash((parent_block_hash, block_tokens, extra_keys)), so each key depends on the whole prefix before it. Prefix caching is on by default with SHA-256 hashing. The BlockPool keeps a flat map from hash to block and a free queue in least-recently-used (LRU) order. A cache hit is the longest chain of full blocks whose hashes are in the map. Only full blocks get a hash. For a model with one attention type, a hit therefore ends on a 16-token boundary. The last partial block of a prompt is not shared.

Hybrid models make this harder. A model with full attention, sliding-window attention and Mamba layers needs different cache rules per layer type. vLLM gives each type its own manager. A hybrid coordinator repeats a fixed-point loop until every manager accepts one common hit length.

SGLang walks a radix tree. The default cache is UnifiedRadixCache, and its tree core now defaults to a Rust implementation, with a Python fallback. Nodes hold token sequences of any length. The default page size is 1 token, so a match can end at any token. Eviction uses LRU by default and skips nodes that running requests have locked. An optional host-memory and storage tier, HiCache, extends the tree.

TensorRT-LLM and LMDeploy sit between. TensorRT-LLM's KV cache manager V2 is the default. It is a C++ radix tree whose keys are chained SHA-256 digests over 32-token blocks. It matches whole blocks, like vLLM, but arranges them in a tree. LMDeploy ships prefix caching off by default. Its PyTorch engine uses a trie of full 64-token blocks and checks the exact tokens on each hit. TurboMind's prefix_trie.h is a flat hash map keyed by length and a cumulative hash, which can match partial blocks.

Fig. 4 shows one effect of the different defaults.

The trade-off is metadata against granularity. A block hash map has one entry per block, and a lookup is a chain of hash probes. That works well with vLLM's per-layer-type managers, because each manager can answer for its own blocks. A token-level tree reuses more of each prompt and keeps branching conversations in one structure. It needs more nodes and a tree walk. TensorRT-LLM's choice shows that the two ideas combine: block-sized keys in a tree.

For a real workload, the difference shows up in the prefix hit rate, not in a feature list. Short shared prefixes, many multi-turn sessions and agent loops that branch from the same context gain the most from finer matching.

TensorRT-LLM and LMDeploy: two other trade-offs

TensorRT-LLM keeps NVIDIA's runtime, not its engine format. With the TensorRT backend removed, a model no longer needs an offline engine build. The install has no trtllm-build command. The runtime is still distinctive:

• Even one GPU uses a proxy plus a separate MPI worker process, "for streaming performance".
• Only MPI rank 0 reads requests. It broadcasts new requests to all ranks each iteration, and every rank schedules the same list.
• Attention DP assigns requests to ranks with a router. One router option scores a rank by uncached prompt tokens plus load, so it prefers ranks that already hold a prefix.
• Orchestrators are MPI by default, or RPC or Ray.

LMDeploy maintains two engines. autoget_backend prefers TurboMind when it supports the model architecture, and falls back to PyTorch. TurboMind runs one C++ engine per GPU as threads in one process. It has no CUDA-graph capture and no speculative decoding. The PyTorch engine has decode CUDA graphs, EAGLE-family and MTP draft models, and prefill/decode disaggregation. When serving, api_server moves the PyTorch engine into a child process reached over ZMQ. Two engines give a fast path and a broad path. The cost is that each feature must be built twice, or it exists in only one engine.

Omni serving changes what a stage is

An omni model is several models in a chain. Qwen3-Omni is the standard example in both omni projects:

1. A Thinker reads text, images and audio, and writes text.
2. A Talker turns that text into audio codec tokens.
3. Code2Wav, a convolutional vocoder, turns codec tokens into sound.

A listener wants the first sound before the Thinker finishes. So each stage must stream partial results to the next stage. The scheduler must now decide where each stage runs, what moves between stages, and when the whole request is done. Fig. 5 compares the two answers.

vLLM-Omni: each stage is a vLLM engine

vLLM-Omni separates the model graph from the deployment. A frozen PipelineConfig holds the stages, their execution types and the functions that convert one stage's output into the next stage's input. A DeployConfig, loaded from a YAML file, holds devices, replicas, connectors and batch limits. The Qwen3-Omni pipeline is Thinker (autoregressive), Talker (autoregressive) and Code2Wav (generation).

Each LLM stage is a full vLLM engine. StageEngineCoreProc subclasses vLLM's EngineCoreProc, and each replica runs in its own process with its own scheduler, KV cache and model runner. The stage schedulers subclass vLLM's Scheduler and AsyncScheduler. An Orchestrator runs on a background thread inside the API process. It tracks each request across stages, handles cancellation and decides when the request is complete.

The default Qwen3-Omni deploy file turns on async chunking. With it, the Orchestrator leaves the data path:

• It pre-submits placeholder requests to the downstream stages. They wait in a WAITING_FOR_CHUNK status, which vLLM-Omni adds to vLLM's request statuses at import time.
• Each producer's scheduler writes chunks into a connector itself. The default connector is POSIX shared memory.
• The Thinker's first chunk carries prompt embeddings, the Talker's hidden-state layer and token IDs. Later chunks carry one token embedding each.
• The Talker sends codec frames to Code2Wav in chunks of 4 frames, then 25, with 25 frames of left context.

Code2Wav is not autoregressive. It runs as a generation stage with a scheduler that subclasses vLLM's synchronous Scheduler, and its deploy entry turns async scheduling off. The Orchestrator forwards only the outputs of final stages, here Thinker text and Code2Wav audio. The API merges them into one stream tagged by modality.

This gives every stage the whole vLLM engine: continuous batching, CUDA graphs, prefix caching and vLLM's model support. The cost is coupling. vLLM-Omni subclasses vLLM's schedulers, workers and runners, and replaces vLLM classes in sys.modules when it is imported. It releases in step with vLLM, 0.32 with 0.32. That pin lives in the installation docs and a runtime warning, not in pip requirements.

SGLang-Omni: stages are nodes in a graph

SGLang-Omni is a separate repository, started in January 2026. A stage is an I/O shell around a pluggable scheduler. Stages declare next, stream_to and wait_for. The Qwen3-Omni speech graph has seven stages: preprocessing, an image encoder, an audio encoder, the Thinker, a text decode stage, the Talker and Code2Wav. Stages that name the same process share one OS process.

A Coordinator sends control messages over ZMQ and merges the results of the terminal stages. A router selects the data path for each pair of stages. It uses an object pass in one process, CUDA IPC on one node, shared memory for CPU data, or Mooncake across nodes. The relay registry also lists NCCL and NIXL, but the router never selects them at v0.2.0.

The autoregressive stages reuse SGLang inside the stage process. OmniScheduler does not inherit from SGLang's Scheduler. It calls the upstream scheduler's methods by composition, and its model runner subclasses SGLang's ModelRunner. SGLang-Omni pins sglang==0.5.22 in pip, one release behind the SGLang tag in this post.

The Thinker-to-Talker hand-off is the opposite of vLLM-Omni's. The SGLang-Omni Thinker streams token-ID tensors, and only the first Talker chunk carries prompt hidden rows. The Talker rebuilds each embedding from the Thinker's embedding table.

The trade-off is clear. vLLM-Omni gives every stage a full engine and pays with deep, version-locked coupling and one heavy process per stage. SGLang-Omni makes encoders, detokenizers and vocoders first-class graph nodes and can pack stages into fewer processes. It pays by owning more of the runtime itself.

Diffusion schedules requests or steps

Image and video diffusion add another unit: the denoising step. SGLang Diffusion (python/sglang/multimodal_gen) is a separate runtime inside the SGLang repository. Its denoising loop runs inside one DenoisingStage call, so one call covers all steps. Dynamic batching can merge compatible requests, but it is off by default and schedules whole requests.

vLLM-Omni's diffusion engine has a request-batch mode and an experimental step-batch mode, which advances compatible requests one denoising step at a time. The diffusion scheduler that now sits in vLLM core is different. Its DiffusionConfig is for discrete diffusion language models, which generate text.

In RL, the engine becomes a rollout worker

RL post-training needs the policy model twice. The trainer holds it sharded for backpropagation, with FSDP or Megatron. The rollout engine holds it laid out for fast generation, with vLLM or SGLang. After each training step, the engine must receive the new weights before it generates again.

Three choices separate the frameworks: where the engine runs, how new weights reach it, and how much lag between the two copies the framework accepts. The table compares them.

Fig. 6 follows one training step in two of these designs.

Sharing GPUs means taking turns

verl uses a single-controller design from HybridFlow. A decorator marks worker methods with dispatch and collect modes. The worker group turns each marked method into a call that fans out to every Ray actor and gathers the results. verl names three rollout modes:

• HYBRID: the engine and trainer share the same GPUs and take turns. All three V1 trainers use it.
• COLOCATED: a separate engine process in the same placement group. verl uses it for reward and teacher models.
• STANDALONE: the engine has its own GPUs.

The vLLM rollout server is a Ray actor that builds AsyncLLM with sleep mode on. In the sync trainer, the step is a hand-over. After sampling, sleep_replicas() calls a level-2 sleep. vLLM 0.32 documents that level 2 discards the weights and the KV cache. The trainer then loads its parameters and optimizer state and trains. At the end of the step, verl wakes only the weight buffers, copies the weights, offloads the trainer and recreates the KV cache. The copy goes through one reusable 2048 MB GPU staging buffer. Its CUDA IPC handle crosses a ZMQ socket once. After that, only bucket metadata crosses.

slime pairs Megatron with SGLang servers. It starts SGLang as an HTTP server subprocess and puts the sglang-router package in front of the servers. On shared GPUs, it releases and resumes SGLang memory with memory-occupation calls. The colocated weight update gathers CUDA IPC handle strings over a Gloo group, and sends them to the local server in an HTTP update_weights_from_tensor call. slime chooses its weight-sync class from flags: disk delta, disk, tensor (colocated) or NCCL.

OpenRLHF wraps vLLM's AsyncLLMEngine, an alias of AsyncLLM, in a Ray actor. It uses CUDA IPC only when all models share GPUs in sync mode, and sends one parameter at a time.

Colocation keeps every GPU busy. Its cost is that generation and training never overlap. The slowest rollout in a batch holds up the whole cluster.

Separate GPUs mean shipping weights and bounding staleness

When the engine has its own GPUs, training can overlap generation. Two problems follow. The weights must travel between GPUs, and some trajectories are generated by an older policy.

AReaL is built around this case. It runs SGLang or vLLM as remote HTTP servers. Its staleness manager limits admission. A new rollout can start only while the capacity formula allows it:

capacity = min(concurrency − running, (η + v + 1) × B − (accepted + running))

Here v is the current policy version, B is the training batch size and η is max_head_offpolicyness, which defaults to 0. With η = 0, AReaL is effectively on-policy, even with separate GPUs. A weight update pauses generation and aborts running requests. AReaL then broadcasts 1024 MB buckets over NCCL. Interrupted requests resend the prompt plus the tokens they already generated, and every token records the policy version that produced it.

verl's separate-async trainer limits consumption instead. It drops a trajectory group, or waits for it, when the group spans more than eight policy versions. Its disaggregated weight transport is a pluggable "checkpoint engine" with NCCL, NIXL, Mooncake, a Kimi peer-to-peer engine and a sparse delta backend. slime enforces no cap. It records staleness and schedules the oldest work first. OpenRLHF bounds the queue depth.

The engine API is ahead of the frameworks

The last row of the RL table matters most for maintenance. No framework here pins vLLM 0.32 or SGLang 0.5.23. verl pins vLLM 0.24.0 and SGLang 0.5.12. AReaL pins SGLang 0.5.10.post1 and vLLM 0.19.1.

The newer engines also change the contract. vLLM 0.32 has a native weight-transfer API with init_weight_transfer_engine, update_weights and a weight version. SGLang 0.5.23 makes tensor and NCCL weight updates fail unless a begin_weight_update session is open. A search finds no call to that session API in verl, slime or AReaL at their tags.

So the RL integration boundary is a version contract. It is not one stable API. A framework that wraps an engine inherits its release cycle, and an upgrade can change how memory release and weight updates work.

Choose by boundary, then measure

A feature table cannot choose an engine for you. The boundaries can tell you what to test. Use this table as a starting point:

Then measure on your own model, hardware and traffic. Report time to first token, inter-token latency and throughput separately, with the settings that produced them. The vLLM article lists the conditions that a fair benchmark report should include.

Key takeaways

• The four LLM engines share one skeleton. Their differences are where text work runs, how many processes a request crosses and where stop strings are checked.
• Overlap scheduling is the default everywhere. It works only while the next step's state stays on the GPU, so speculative decoding and grammars need special handling.
• vLLM indexes prefixes as hashed blocks, and SGLang uses a token-level radix tree. TensorRT-LLM combines block keys with a tree.
• vLLM-Omni builds each omni stage from a full vLLM engine. SGLang-Omni builds a coordinator graph that borrows SGLang's scheduler. Both stream between stages.
• RL frameworks embed the same engines with different placement, weight transport and staleness rules. They pin older engine releases than the newest engine APIs.

Further reading

• LLM Inference Is a Full-Stack Systems Problem explains prefill, decode, kernels and scheduling.
• The Life of a Request Inside vLLM explains continuous batching, PagedAttention and prefix caching.

References

1. vLLM Project, vLLM v0.32.0, October 2026.
2. SGLang Project, SGLang v0.5.23, November 2026.
3. NVIDIA, TensorRT-LLM v1.4.0rc2, October 2026.
4. InternLM, LMDeploy v0.18.0, September 2026.
5. vLLM Project, vLLM-Omni v0.32.0, October 2026.
6. SGLang Project, SGLang-Omni v0.2.0, October 2026.
7. verl Project, verl v0.9.1, September 2026.
8. THUDM, slime v0.4.0, October 2026.
9. OpenRLHF, OpenRLHF v0.11.2, September 2026.
10. AReaL Project, AReaL v2.1.0, August 2026.
11. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.
12. Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, NeurIPS 2024.
13. Sheng et al., HybridFlow: A Flexible and Efficient RLHF Framework, EuroSys 2025.
14. Fu et al., AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning, 2025.

Note: This blog was researched and drafted with the assistance of Claude Code (Claude Opus 5.5).

Updated 03 November 2026