BLOG ILLUSTRATION / EXECUTION STORY

One RL step hands weights to the rollout engine

Two source-derived traces follow one RL training step. In verl's colocated sync mode, the engine and trainer take turns on the same GPUs. In AReaL's disaggregated mode, training overlaps generation and new weights interrupt running requests.

One RL step hands weights to the rollout engine A controller sends prompts to a rollout engine and can also sleep or pause it. The engine writes trajectories to a replay buffer. The trainer reads a batch from the buffer and sends new weights back to the engine. Controller training loop Rollout engine vLLM or SGLang Replay buffer trajectories Trainer FSDP or Megatron prompts sleeps or pauses trajectories batch new weights Solid: data Dashed: control
Author-created trace from verl v0.9.1 (HYBRID mode, vLLM rollout) and AReaL v2.1.0 (xccl weight updates). Function names follow the pinned source. Playback time is not measured latency or memory use. Open full-size SVG