BLOG ILLUSTRATION / EXECUTION STORY

Dreamer learning and actor deployment

World-model fitting uses replay to infer latent state and fit prediction heads. Behavior learning starts from replay states. Actor and dynamics give imagined trajectories for return estimation and actor/critic updates. A different deployment path infers the environment state and samples the actor without lookahead search.

Dreamer learning and actor deploymentWorld-model fitting uses replay to infer latent state and fit prediction heads. Behavior learning starts from replay states. Actor and dynamics give imagined trajectories for return estimation and actor/critic updates. A different deployment path infers the environment state and samples the actor without lookahead search. World-model fitting Behavior learning in imagination Actor deployment Replay Observation + action Infer latent state Posterior state Model fitting Prediction heads Replay state Posterior seed Imagine Actor + dynamics Returns Reward + value Actor + critic Behavior update New observation Infer latent state Learned model Actor sampling No lookahead search Policy update
Dreamer trains the world model from replay, then trains actor and critic on imagined trajectories that start from replay states. Deployment samples the actor after state inference without lookahead search. Author-redrawn mechanism from Hafner et al., without reconstruction images and detailed loss/gradient paths. Source: Hafner et al. Playback time is illustrative and is not measured runtime. Open full-size SVG