A robot arm prepares to move an orange cube into a tray beside a black-outline sketch of the proposed outcome. — A predicted future matters to a robot when it can guide a concrete action and be checked against the resulting scene.

Embodied AI World Models

Wenbo Ji 09 June 2026 46 min read

In brief

A robot world model predicts results for a planner, evaluator or learning algorithm. For control, a connection to executable actions and tests of uncertainty and recovery are necessary. Video models, latent dynamics models and joint world-action models have different functions in this loop.

World-model families have different prediction targets, action inputs and planning processes.
Robot datasets must align observations, actions, timing, rewards and embodiment metadata.
Do checks of uncertainty and model exploitation. During planning with predicted futures, use new observations to replan as drift increases.

Predictive models can help robots only through connections to action, uncertainty and deployment.

A robot predicts that a pull will open a drawer, but its gripper slips. The prediction helps control only if the system can use the observed change to select its next command.

The term “world model” can refer to different parts of that problem.

In reinforcement learning (RL), the term can refer to learned dynamics for imagined rollouts. A video model can predict possible future frames. A robotics model can represent objects, geometry, contacts and action results. A driving model can simulate traffic. Foundation-model researchers also use the term for general visual knowledge. This knowledge can help with behavior that has some properties of planning.

I compare these methods with three questions:

Which quantities does the model predict? Which inputs condition that prediction? Which decision uses the result?

These questions apply to predictions of passive frames, actions, value distributions and joint trajectories.

Yang Li’s world model tutorial gives an introduction to the general concept. This post examines how predictions help an agent select actions with partial observations, delayed computation, contact and hardware constraints.

Start with one interaction:

o_t,\ a_t \rightarrow o_{t+1}

The robot gets an observation, executes an action and gets a new observation. Alternatives include pixels or latent states, motor commands or action chunks, and one step or a long rollout. Data can come from passive video or robot episodes. The system can use imagination during training or planning during execution.

Fig. 1. A world model receives context and a candidate robot action. Its predicted results enter a planner, evaluator, filter or policy learner. The executed action returns new observations. Full interactive figure

The path from prediction to control

The remaining sections examine four parts of the drawer task. They examine which state perception must keep and how a prediction changes a decision. Then they examine the mapping from visual change to robot commands, and the necessary data and timing.

Connections between these research areas

World models are not new. Research in model-based reinforcement learning started decades before this post and continues to try to learn environment dynamics. In 2018, World Models made the "train inside a dream" idea known to more deep-learning researchers. Different research areas use the same term for related purposes.

Video generation is one research area. Consistent objects, approximate geometry and possible motion suggest that a model learns more than texture. Video contains object permanence, agent motion, signs of contact, lighting and layout. These patterns can help, but tests of controllable dynamics are also necessary.

Robot data gives one more reason for this research. Vision-language-action (VLA) models map camera observations and instructions to commands. They use trajectories with action labels, and collection has high costs. Teleoperation is slow, hardware time is limited, and failures can cause damage. Each robot body has a set of actions for that body. Predictive learning uses visual data, robot play, simulation, demonstrations and deployment logs with the goal of better policy data efficiency.

The third research area is autonomous driving and closed-loop evaluation. Driving systems use counterfactual simulation to compare alternative actions. For example, which result does the model predict for earlier acceleration, a different merge, or earlier braking? Researchers cannot collect rare safety cases on demand. A learned world model can generate scenarios, simulate sensors or evaluate plans. These plans can be too dangerous or too rare for direct road tests.

Infrastructure adds more possible functions: data generation, evaluation services, synthetic experience, representation learning and action filtering. Thus, RL research, robot development plans, driving systems and video demonstrations use the same name.

The shared vocabulary refers to different functions. Different checks are necessary for a latent predictor for training, a search simulator and a video prior for action learning.

From observations to a predictive state

In the most general definition, a world model predicts change. Control theory expresses the interaction as:

s_t,\ a_t \rightarrow s_{t+1}

The agent is in state $s_t$ . It executes action $a_t$ . The world changes to the next state $s_{t+1}$ .

In robotics, the full state and exact dynamics are rarely available.

A state is the full set of variables necessary to represent the situation sufficiently for control. These variables include object poses, velocities, robot joint positions, gripper state, contact state, hidden forces and, in some cases, task progress. The real state is usually not available.

An observation is the information that the robot receives. It can include camera frames, depth maps, proprioception, tactile readings, language instructions and partial object state. Proprioception is the robot's internal measurements, such as joint angles, joint velocities, end-effector pose and gripper opening. These signals frequently change the success of conversion from a visual prediction to a robot action.

An action is a command for the robot or controller. Examples include joint position, joint velocity, end-effector displacement and gripper opening. An action chunk gives predictions for multiple commands at the same time. Its interpretation changes with embodiment: the body, sensors and action space. A Franka arm, suction gripper, mobile manipulator and human hand can cause visual changes with some of the same properties. Their commands are different.

Thus, robot world models frequently operate in observation space or latent space:

p(o_{t+1} \mid o_t, a_t)

The model receives the current observation $o_t$ and action $a_t$ . It predicts a distribution of possible next observations $o_{t+1}$ .

One image rarely contains sufficient state information. A previous view can give information about a hidden handle. The history of a pull can give information about a resistant drawer. This tutorial uses predictive state $b_t$ to represent that history:

h_t=(o_{\leq t},a_{<t}),\qquad b_t=E_\theta(h_t),\qquad p_\theta(b_{t+1},r_t\mid b_t,a_t).

Here $h_t$ is the observation-action history, $E_\theta$ represents it, and $r_t$ is an optional task reward. I use this notation to compare methods. Recurrent latents, object maps or video-context caches can make $b_t$ . But two different hidden contact states can have the same representation.

Representation learning selects the information that $b_t$ keeps. Dynamics learning predicts results for action $a_t$ , and a policy or planner uses those results. Tests of each selection are necessary. A model can predict frames accurately, but a robot can select an incorrect action.

A mug can tip or stay upright. Cloth can fold or bunch, and a person can walk or stop. A distribution represents these alternatives. I recommend control decisions that include contact uncertainty, without an assumption of a known future.

In this example, a robot tries to pull a drawer open. At time $t$ , $o_t$ can include a wrist-camera image of the handle and a third-person camera view. It can also include the robot joint positions and gripper opening. The action $a_t$ can be a short action chunk: "move the end effector 2 cm forward, close the gripper, then pull 3 cm back". The next observation $o_{t+1}$ possibly shows the gripper attached to the handle and the drawer open by a small distance.

The transition connects a command to visual and proprioceptive change for this body and calibration. Calibration maps camera pixels, robot coordinates and physical space. A calibration error can cause a grasp at the incorrect location, although the grasp looks possible.

If the next frame shows a gripper slip, I recommend two responses from the world model. It predicts a different future and decreases its confidence in the same pull. A closed-loop robot then gets a new observation and makes a new estimate of the handle pose. It possibly adjusts the wrist angle and tries a safer local action.

This process is closed-loop control: the robot executes commands, gets observations and corrects errors. The process occurs again as real execution causes errors.

I recommend three different definitions.

In the broad definition, a world model predicts future structure from context. Video/text models, driving sensor simulators and game dynamics models learn patterns in their domains and extrapolate from them.

The robotics and RL definition conditions predictions on actions. Planning, counterfactual reasoning and model-based policy learning examine the result of a specified intervention, not only the usual next event.

An action-conditioned likelihood alone does not guarantee causal identification. Hidden state or the data-collection policy can confound action with outcome. Then the model can learn a conditional correlation that fails for a new intervention. Executable actions and aligned episodes are the only requirements in this tutorial. A different test is necessary for the validity of the model's counterfactuals for new interventions.

A World Action Model (WAM) couples world prediction and action prediction. A policy maps $o_t, \ell \rightarrow a_t$ . A task-conditioned video predictor maps $o_t, \ell \rightarrow o_{t+1:T}$ . A WAM learns future world trajectories and action trajectories together.

Static interface table: forward models predict results, inverse models infer actions, policies select commands and task goals condition futures. — Fig. 2. Forward dynamics predicts action results, and inverse modeling infers actions from observed change. Policies output actions from task context, and task-conditioned predictors output desired or plausible futures.

With observations and actions, the interface has multiple related forms.

Forward observation prediction:

o_t, a_t \rightarrow o_{t+1}

Inverse action inference:

o_t, o_{t+1} \rightarrow a_t

Policy learning:

o_t, \ell \rightarrow a_t

Future prediction conditioned on a task:

o_t, \ell \rightarrow o_{t+1:T}

Here $\ell$ is a language instruction or task description, and $o_{t+1:T}$ is a future observation sequence.

The formulas express different functions. Forward models predict a command's result. Inverse models infer commands from change. Policies select the next command. Task-conditioned predictors represent progress to a goal.

WAMs connect future prediction to action generation. Independent video and control modules are not a requirement.

I use these categories to select measurements or systems to make. For control, predictions in observation space or state space must have a defined connection to executable actions.

Compare prediction targets, actions and functions

Current world-model research includes related approaches: compact latent dynamics, pixel prediction, hidden states for decisions, latent representation prediction, interactive video and large video priors. A large video prior does not necessarily recover robot actions.

Static method atlas: six world-model families have different prediction targets, action inputs and learners or planners. — Fig. 3. The atlas organizes selected families by prediction target, action input and the component that usually uses the prediction. Its columns compare inputs and outputs.

The first classification is the prediction target.

First, pixels or video. A predicted video of block pushing is easy to examine. But it also models shadows, texture, noise, background motion and lighting, which do not necessarily change control decisions.

Second, latent states. Dynamics learning in compressed hidden variables usually costs less than pixel prediction. But compression can remove details that affect action selection.

Third, structured state. Some systems represent objects, geometry, contacts, occupancy, depth, semantic maps or robot proprioception. These representations are less general than raw video models, but planning with them can be easier.

Fourth, 3D or 4D worlds. For robots and driving, a future video from one camera is frequently not sufficient. Necessary spatial predictions include: the object location, possible gripper collisions and contact stability. Multiple views, depth, point clouds and scene representations can supply this information.

Fifth, reward, value, or task progress. Some world models predict variables without a visual rendering. For example, MuZero models the quantities necessary for search: latent transitions, reward, policy and value. Its predictions give information for decisions, not a future image.

Sixth, actions. WAM-style systems learn future world evolution and its action sequence together. Recovery of the commands by a different inverse model is not necessary.

A learned world model does not replace a physics engine. Simulators have advantages when geometry, contacts and materials are known. Learned models have advantages when data is available but the full physical description is not. Many systems mix the two approaches.

For robotics, the primary test is:

Does an action that the robot can execute condition the predicted future?

For control, an approximate prediction for feasible commands can help more than a visually attractive video without action grounding.

The second classification is how action enters:

Passive prediction: the model receives context and predicts the usual next event. This can help with visual priors and representation learning, but gives limited information for counterfactual control.
Action-conditioned prediction: the model receives an action or action sequence and predicts the future. Planning, model predictive control and policy evaluation can use this prediction.
Latent-action prediction: the model learns action-like variables from unlabeled video. This can help a system use internet video or human video. But a mapping from the latent action to a robot controller is necessary.
Direct action generation: the model predicts the action, as in a VLA policy.
Joint world-action modeling: the model predicts future world state and future action together, as in WAM-style systems.

The third classification is where the model is used:

Representation learning: the model learns features for easier downstream control.
Planner: the planner simulates candidate actions and selects the action with the best predicted result.
Evaluator: the evaluator scores the probability of success for a policy, plan or subgoal.
Action filter: the filter rejects actions with uncertain, implausible or dangerous predicted futures.
Training simulator: the model generates imagined rollouts, synthetic episodes or future-prediction losses for policy learning.
Subgoal generator: the model predicts a near-future state as a target for a lower-level policy.

Before you compare results, identify each paper's predicted output, conditioning inputs and the component that uses the prediction.

I use this checklist to read papers:

Prediction target: is the output a pixel frame, latent state, 3D state, reward/value, action, or a combination of these?
Action grounding: does the model receive executable robot actions, latent actions from video, language goals, or no action at all?
Time horizon: is it trained for one-step prediction, short-horizon planning, long rollout, or test-time future imagination?
Data tuple: can you write the training example as $(o_t, a_t, o_{t+1})$ , $(o_t, \ell, a_t)$ , $(o_t, \ell, o_{t+1:T})$ , or a different tuple?
User of the prediction: does a policy, planner, evaluator, safety filter or representation learner use the prediction? Or does the system only display generated media?
Failure mode: if the model is incorrect, does the system only generate an incorrect video? Or does it select an incorrect action, cause hardware damage, or train a brittle policy without an error signal?

For robotics, I usually start with items 2 and 6. They identify the connection to executable commands and the failures that an incorrect model causes. These answers usually identify a paper that can help with implementation at this time. A passive video model can give a prior that helps. But a method to identify prediction drift is also necessary.

Which state must compression keep?

A model can represent the same drawer as a rendered image, a hidden code or a structured object with pose and contact state. The correct representation changes with the errors that affect action selection. Removal of wood grain can help. Removal of the small space that shows an ungrasped handle can cause failure. The test for compression is preservation of the details that change action selection.

Memory in a compact state

The classic World Models paper gives a design with three parts. The system compresses raw observations, predicts latent transitions and optimizes control in this compact belief space.

Fig. 4. In the classic V–M–C design, the controller reads current visual code z_t and memory h_t to select a_t. The recurrent memory then updates from z_t and a_t. The controller does not directly receive the memory model's next-code prediction. Full interactive figure

The architecture has three parts:

A vision model compresses observations into a latent code.
A memory model predicts how the latent state evolves.
A controller selects actions from the compact internal state.

The design has different components for compact state, dynamics and control. A variational autoencoder (VAE) and recurrent neural network (RNN) are one implementation, not necessary components. Compression supplies state for prediction and action selection. Video generation by the controller is not necessary.

The information available to the internal simulator comes from the representation and loss. Information not in the representation or the loss is a model limitation.

This information is possibly sufficient for a game agent. It is not necessarily sufficient for a robot in contact with deformable objects.

Which information does the prediction loss keep?

Prediction targets shape the state. Embedding prediction can remove appearance details. Tests are necessary to find if that removal helps control.

A Joint-Embedding Predictive Architecture (JEPA) predicts representations of hidden input regions, not their pixels. V-JEPA applies embedding prediction to video.

Fig. 5. V-JEPA learns to predict held-out target embeddings from visible video context. The target encoder uses an exponential moving average (EMA), and its features are stop-gradient targets. Author-redrawn mechanism from Bardes et al. Source: Bardes et al. Full interactive figure

For control, the goal is to keep changes that affect actions with less appearance modeling. The learned targets specify the information that the representation keeps.

For the drawer, a representation can keep sliding motion, a graspable handle and changes in free space, but discard wood grain. Latent prediction lets a model make this selection.

General visual prediction or object recognition can discard contact cues. Features for possible interactions (affordances), motion constraints and action results are necessary for a robot.

Action conditioning connects the next state to the robot's command.

The distinction between reconstruction and embedding has a counterexample. DreamerV3's learning-signal ablations stop reconstruction gradients or task-specific reward/value gradients on the path to representation learning. In the reported task set, reconstruction is a primary source of performance.

In contrast, TD-MPC2 learns without a decoder. It combines latent prediction with reward learning and value learning. A prediction target helps through the information it keeps and the resulting behavior. Removal of pixel reconstruction alone does not give a better state.

Design alternative	Target information for the state	Necessary robot tests
Observation reconstruction and latent dynamics prediction	Sensory structure plus temporal predictability, as in Dreamer	Does the state keep contact cues and let the model imagine behavior that helps the task?
Latent transition, reward and value prediction without rendering	Results necessary to score action sequences, as in TD-MPC	Is action ranking correct on tasks outside the training set?
Transition, reward, policy and value learning for search	Information necessary for search, as in MuZero	Does the search interface agree with the robot's action space and timing?
Prediction of learned visual embeddings	Structure in the target representation, as in V-JEPA	Does that structure contain results of executable actions?

My interpretation of these objectives is a question: which information can the model discard without a different decision? Reconstruction can keep details that help. Task losses focus capacity. Each objective can miss a hidden force or an unseen failure.

How does a predicted future change a decision?

The same latent model can train a policy, supply a planner with predictions during execution, or supply predictions for search. These functions use computation at different times and have different failure modes.

Train behavior with imagined results

DreamerV3-style agents use latent dynamics to train behavior. They extend PlaNet, which introduced planning in a learned recurrent latent state space. Dreamer trains behavior fully in imagined latent trajectories. This training decreases the necessary number of real-world interactions.

Fig. 6. Dreamer trains the world model from replay, then trains actor and critic on imagined trajectories that start from replay states. Deployment samples the actor after state inference without lookahead search. Author-redrawn mechanism from Hafner et al. Source: Hafner et al. Full interactive figure

Imagined learning can decrease the number of physical trials with high costs.

The related risk is model exploitation. A policy finds actions with high scores only because the learned model is incorrect. For a robot, the result can be a bent gripper, a dropped object or a predicted success that fails at contact.

Dreamer has different processes for model learning and behavior learning. The world model encodes observations and predicts transitions, reward and continuation. Dreamer’s actor (policy) and critic (value estimator) learn through imagined latent trajectories. These different processes let researchers change the representation or imagination horizon and deploy a faster policy. These processes also let researchers make different analyses of perception, dynamics, reward or exploitation failures.

In DreamerV3's algorithm, imagined trajectories train the actor and critic. During environment interaction, the system samples the actor without lookahead planning. The model also helps infer the current recurrent state. For deployment, find the differences between these functions. State inference with a learned model is different from search through imagined futures before each command.

Compare action sequences at execution time

TD-MPC and TD-MPC2 learn a task-oriented dynamics model, without a requirement for convincing future rendering. They evaluate candidate action sequences through short-horizon model predictive control (MPC).

MPC compares short futures and executes a small part of the selected sequence. Then it gets a new observation and corrects the plan:

observe -> infer latent state -> sample candidate action sequences
        -> roll out short latent futures -> score with reward/value
        -> execute first action or chunk -> observe again

TD-MPC2 includes state/pixel observations, single/multi-task training and online RL/offline data. Its domains include DMControl, Meta-World, ManiSkill2 and MyoSuite. Its compact task-oriented dynamics show a method to scale a control model without photorealistic generation.

A usual expression for the planner's scoring problem is:

\hat J(a_{t:t+H-1};b_t) =\mathbb E_\theta\!\left[\sum_{k=0}^{H-1}\gamma^k\hat r_{t+k} +\gamma^H\hat V(b_{t+H})\right].

Here $H$ is the local planning horizon, and $\gamma$ discounts later reward. The terminal value estimates results after that horizon. This formula is a schematic MPC objective. The formula shows the difference from actor training. The planner uses model computation to compare candidates during execution. The actor learned a mapping from state to action with lower computational cost before execution.

For each method, the learned predictions must rank actions that help the task correctly.

Keep the quantities necessary for search

MuZero shows that a world model can fully ignore visual reconstruction. It learns only the latent transitions, rewards, policy and values necessary for planning search.

Fig. 7. MuZero uses h for history encoding, g for latent transitions and rewards, and f for policy/value predictions in search. After environment actions, h encodes new observations. Author-redrawn mechanism from Schrittwieser et al. Source: Schrittwieser et al. Full interactive figure

For a robot, object pose, gripper opening, contact state and slippery-surface uncertainty can be more important than the table's color. Prediction must keep the variables that change action selection.

Identification of these details before a decision is not easy. A learned latent model can discard nuisance variation. But it can also discard a small visual cue. Without that cue, the grasp can fail.

For me, MuZero is an example of prediction for decisions. Visual quality and control quality can be different.

The shared test is decision quality. An actor can learn errors from its training simulator. MPC can select actions that exploit short-horizon prediction errors again and again. Search can use an incorrect value as if it were correct.

After the drawer gripper slips, I think that correct action ranking helps more than a detailed video of a successful pull. The ranking gives a small wrist correction a better score than one more forceful pull.

How does visual change become an executable action?

The next problem is action grounding. A visual prior can represent the usual drawer motion without a mapping to the robot command that causes this motion.

Video models show future predictions. An interactive generator also lets a latent action change the predicted future. This function suggests a method for a controllable simulator.

Fig. 8. Genie trains action-conditioned dynamics with video tokens and latent actions inferred from unlabeled transitions. These latent actions are not calibrated robot commands. Author-redrawn mechanism from Bruce et al. Source: Bruce et al. Full interactive figure

Can these visual dynamics give a robot a prior? Action grounding limits the possible transfer.

Internet video rarely records calibrated robot state, exact gripper commands, force feedback or intervention labels. A model can learn that a hand moves a cup. It can do this without a mapping to the robot command for that motion.

Genie-style research tries to learn controllability from unlabeled videos. The model receives transitions and learns latent actions for the change between frames. In a game-like environment, these latent actions can become an interactive interface. Human-labeled button presses are not necessary for each video. The model can learn a small action space that changes the simulated world.

UniSim applies a related idea to a more general simulator. It combines data sources with different advantages. Internet data supplies visual diversity, and robotics data supplies dense action grounding. Navigation or interaction data supplies longer-horizon experience. The goal is more than video synthesis. An interactive environment lets a planner or policy train, search or evaluate predicted results.

Cosmos applies this idea to industrial systems. World models are part of a physical AI data system. They predict futures, transfer domains, generate synthetic experience, connect to simulation tools and help train robot models or driving models. The model is more than a network checkpoint. It is an infrastructure component for data generation, evaluation and deployment.

Scale supplies visual situations that a laboratory's teleoperation budget cannot collect. Grounding must connect mug or drawer motion to gripper force, wrist rotation and end-effector commands.

This is where inverse action enters:

o_t, o_{t+1} \rightarrow a_t

Action inference helps a system use unlabeled video. A mapping from a latent “move object left” to commands for the specified body, camera, gripper and controller is necessary.

This mapping has multiple problems. Action is embodiment-specific. A human hand, two-finger gripper, suction cup and mobile manipulator can cause visual changes with some of the same properties. Their commands are different.

Camera motion is not controller state. Two videos can show the same object movement. But the robot action can have coordinates in joint space, end-effector space or an action-token space.

The time scales are different. Web video has edits and compression, but high-frequency contact detail is necessary for robot controllers. The data does not include sufficient failures. Examples of partial grasps, slips, collisions, stuck drawers and recovery attempts are necessary, not only successful videos.

Couple future observation and action learning

World Action Models, or WAMs, connect video prediction and robot policy learning. Their premise is that, in some cases, joint learning of future visual change and future motor commands is better.

The OpenMOSS Awesome-WAM survey distinguishes three conditional distributions:

\text{VLA:}\quad p(a \mid o, \ell)

\text{WM:}\quad p(o' \mid o, a)

\text{WAM:}\quad p(o', a \mid o, \ell)

A VLA can map visual-language context directly to action chunks. It can give good results with sufficient data coverage and reaction speed. It does not have to output predicted results of actions.

In contrast, a video predictor alone does not have the direct action grounding necessary to guide a robot. It predicts futures without the motor commands to produce them.

A WAM tries to make future prediction and action learning help each other. This coupling can use generated futures during inference or shared representation learning during training. A future-prediction head helps only if it changes action learning or decision quality. Removal of that head at deployment does not prove that its training signal had no function.

One tuple that you can use for that coupling is:

(o_t, \ell, a_t, o_{t+1}, a_{t+1})

This example tuple connects task, current scene, change and action.

Fig. 9. A joint WAM learns future-world trajectories and action trajectories from observation and task context. Some systems use imagined futures, but others deploy a faster action path shaped by prediction supervision. Full interactive figure

A WAM has multiple possible functions:

Imagine then execute: the system predicts a possible future video, then converts that visual future into actions.
Joint modeling: the system trains future visual states and future actions together.
Training-time regularization: the system learns better robot representations, although deployment uses a faster policy.
Verification: the system evaluates the plausibility of candidate-action futures and their agreement with the task.

Coupling between future prediction and action generation

Compare how the two predictions connect.

A Cascaded WAM predicts a future state or plan first. Then a different model or head converts that future into executable action. Its inputs are observation and language. Its intermediate output is a future state or plan, and its last output is executable action.

A cascade lets researchers examine video, flow, depth, 4D structure, latent plans or subgoals before action decoding. Stages can disagree. Execution of a future that agrees with the task is not always easy. The added stage can also increase latency.

A Joint WAM predicts future state and action in one shared model. It receives observation and language, and outputs future state and action together.

Joint supervision couples the tasks more tightly, but makes optimization less easy. Visual detail can use model capacity, imitation can overfit, and the losses can interfere. Cascades are easier to examine. Joint models can connect action more directly to prediction, but can be less easy to train and debug.

I think that WAMs can help, but I identify timing as their deployment question. Where can future imagination operate without a latency greater than the control budget?

Fast-WAM uses controlled comparisons to examine these processes in a shared implementation. It keeps video co-training. Action tokens cannot read future video tokens. Inference does not include the future-video branch.

Its controlled comparisons include LIBERO, RoboTwin and the reported real-robot tasks. Removal of video co-training causes a larger penalty than removal of explicit test-time imagination in these comparisons. From this result, I think that prediction supervision can help without an inference-time video in these settings.

I recommend different analyses of three claims in WAM results:

Prediction supervision changed the representation or policy during training.
The system generated and used future imagination before an action at inference.
Action evaluation used predicted consequences to compare or reject candidate commands.

A joint training distribution $p(o',a\mid o,\ell)$ does not specify the inference path that the system executes. The paper's masks, losses, sampling procedure and ablations specify that path. My criterion for this coupling is better action generalization, recovery, evaluation or data efficiency in the actual deployment budget.

An ablation is also necessary for the training question. Video supervision and action supervision can help each other. But visual-detail costs and imitation overfit can cause interference. The WAM comparison in the project section compares action-only, future-only and joint objectives. Then it does a test of explicit imagination with the shared training signal.

Data for learning action results

The training examples limit the action results that the model can learn.

For robot learning, the minimal transition tuple is:

(o_t, a_t, o_{t+1})

This tuple gives more information than passive video. It identifies the action associated with a change in observation. But more information is frequently necessary for a robot dataset:

Language instruction
Robot state
Multiple camera views
Task success labels
Reward or preference signals
Human interventions
Episode boundaries
Embodiment metadata
Controller timing

Static data comparison: video, simulation, robot play, demonstrations and interventions supply complementary supervision with different limitations. — Fig. 10. Data sources supply complementary supervision: visual diversity, privileged simulation state, aligned robot commands, expert behavior and failure/recovery evidence. The mixture changes with the component that uses the prediction.

Aligned robot actions are necessary for WAMs. Other sources can also supply diversity, privileged simulation state and visual priors.

Robot-centric teleoperation gives the cleanest aligned state-action supervision. It records robot observations, actions, kinematics, proprioception, timing and frequently contact-rich behavior for the actual embodiment. These data let a WAM connect future prediction to executable commands. But collection is slow. Teleoperation has high costs and hardware limits. Its coverage is usually limited compared with internet video.

Portable human demonstrations are an intermediate source between robot data and web video. Handheld systems or human-demonstration pipelines can collect diverse real-world manipulation at lower cost. They show real objects, clutter, dexterity and long-tail situations. Mapping human behavior to robot action remains necessary. A hand trajectory must become commands for the specified gripper, controller and calibration.

Simulation gives privileged supervision that is rarely available without errors in the real world. Examples include object pose, depth, segmentation, collision state, contact and, in some cases, material parameters. These data help with dynamics learning and failure diagnosis. But simulation has a reality gap. A model can predict simulated contact accurately but miss friction, compliance, lighting, sensor noise or small mechanical delays on the real robot.

Human and egocentric video gives scale and open-world priors. It shows activities, object affordances, long-horizon temporal structure and human interaction patterns that a robot laboratory cannot fully collect. Its grounding for executable action is limited. But a model can learn possible future changes from this video.

Robot play, expert demonstrations, failures, interventions and deployment logs show the successes and failures of the skill. Play data gives coverage. Expert demonstrations give task-aligned behavior. Real deployment identifies the imagined futures that agreed with physical execution.

Failed trajectories show near misses, contact errors and states with necessary recovery. Successful trajectories alone can leave a policy without examples of early failure.

Intervention data can be very important. It identifies the time when an autonomous policy failed and a person corrected it.

No one source is sufficient. Internet video without robot actions has wide coverage but limited action grounding. Expert demonstrations without failures can produce brittle policies. Real deployment data can help, but cost and safety limit collection.

The necessary data mixture changes with the world model's function.

For representation learning, video with wide coverage and unlabeled robot logs can help. For action-conditioned planning, accurate action labels, controller timing and next-state observations are necessary. For safety filtering, rare failures and interventions help more than one thousand added successful pick-and-place trajectories. Close alignment of future observations and future actions is necessary for WAM-style joint modeling. The model must learn their connection, not only two captions without a connection.

Before more data collection, identify the missing tuple.

A transition with more information keeps proprioception, action timing, task outcome, intervention flags, calibration and embodiment attached to the visual change. The full sequence is as important as each tuple.

Keep the episode, timing and embodiment

The data format is an important part of robot learning.

Robot data records ordered interactions with episode boundaries. Controller conventions and robot state give actions their meaning. Rewards and success have a relation to full trajectories and to frames.

Reinforcement Learning Datasets (RLDS) organizes episodes into timesteps. These timesteps include observations, actions, rewards, discounts and metadata.

Episode map: observations, actions, rewards and terminal fields have a shared sequence, task, timing, calibration and robot context. — Fig. 11. An episode keeps ordered timesteps and the task, timing, calibration and embodiment context necessary for action interpretation. Action alignment must identify the observation that the action is intended to affect. Checks of reward, discount and boundary conventions in the dataset that you use are necessary.

A world model trained from these data can learn more than visual continuity. It can learn how action changes the scene and which action sequences are related to success. It can also learn episode termination and the robot or task that produced the data.

Metadata can change the learning target. For one frame, the necessary commands can be different for a two-finger gripper, suction cup and mobile manipulator. Table height, object scale and body also change the trajectory for the same instruction.

Better preservation of these differences decreases the information that the model must infer from limited visual cues.

Open X-Embodiment showed this problem at scale. When many laboratories supply robot datasets, model architecture is only one problem. Schema alignment is one more. Which quantity does an action represent? Is it joint position, joint velocity, end-effector delta, gripper command or a chunked trajectory?

The schema must also identify the wrist-mounted camera and the frame that the action is intended to affect. Does the language instruction refer to the full episode or the current subtask? Does the dataset include failed attempts, or does filtering remove them?

Fields attached to their timestep and episode help in three ways.

First, they keep temporal alignment. I recommend a pairing of the action at timestep $t$ with its subsequent observation. A shuffled frame with some of the same visual properties is not an equivalent pairing.

Second, they keep termination and success context. One transition is near the end of a successful episode. A transition in a failed episode can have some of the same visual properties. But these two transitions have different meanings.

Third, they keep embodiment metadata. A cross-robot model must identify the trajectory's source: a Franka arm, xArm, Google Robot, mobile manipulator or a different platform. Without this context, the model can average incompatible action conventions.

Conditions for closed-loop operation

Contact increases the effects of small errors

Robotic world models connect video prediction to physical action. A blurred frame causes no physical damage, but an incorrect prediction in physical space can cause a collision.

Fig. 12. I recommend execution of only a trusted short rollout prefix, with new observations and replanning as model uncertainty or drift becomes important. The branching fan illustrates prediction ambiguity. Full interactive figure

Manipulation is not easy because task success frequently changes with small events.

A millimeter of gripper offset changes a grasp. A cable catches on an edge. A cloth fold hides part of the state. A drawer sticks. A block turns through a small angle and changes the next contact. These events have small visual effects, but can change task success.

A closed-loop model must let the system replan from new observations and identify rollout drift. It must keep actions feasible and prevent rewards for hallucinated results.

These robotics details make this less easy than ordinary future prediction:

Contact is discontinuous: a small pose error can change a grasp into a push. It can also change an insertion into a jam, or a lift into a slip.
Deformable objects hide state: cloth, rope, food, bags and cables do not have one simple pose. Occlusion can hide an important fold or loop.
Occlusion changes belief: the robot frequently executes commands while the gripper blocks the camera or the object moves behind a different object.
Calibration and embodiment are important: the same image implies different actions for different cameras, grippers, action conventions and low-level controllers.
Latency is part of the model: learned dynamics can be locally reasonable. But observations from a previous state can cause incorrect planning.
Closed-loop recovery is important: real manipulation succeeds through correction, not one long imagined video without feedback.

I recommend a minimum of four questions for a robot world model:

Which changes are probable after this action?
Which parts of the prediction have uncertainty?
Does the predicted change help the task?
Is the imagined rollout sufficiently reliable for the next action?

The last question concerns deployment. A world model can give a good training signal but be too slow or too uncertain for test-time planning. Some systems use future prediction during training to change representations and policies. Then they run a faster policy at deployment. Other systems keep imagination in the control loop. But they limit its horizon or use it as a verifier, not the only controller.

Research on uncertainty and rollout drift examines this deployment problem. For example, Persistent Robot World Models tries to stabilize multi-step robot rollouts, not only make plausible one-step predictions. Model exploitation is a known failure mode in model-based RL. The policy finds action sequences with high scores because the learned model is incorrect for those sequences. In robotics, this error can cause dangerous behavior or behavior that does not help the task.

Possible responses include penalties for high-uncertainty rollouts, training on multi-step self-generated predictions and frequent replanning. These methods try to prevent policy failures in areas where the model is incorrect.

I recommend lower model confidence for states farther from observed data. I also recommend a controller that identifies this lower confidence.

Put the model in a timed deployment loop

A deployed world model operates with perception, planning, safety checks, action filtering, control, monitoring and recovery.

Fig. 13. In this planner example, a world model receives current state and candidate actions, and predicted results enter planning. Feasibility checks occur before timed controller execution. New observations supply feedback, and monitoring and measured evaluation have different functions. Full interactive figure

Each function has a different process. A planner searches, an evaluator scores, a filter rejects, and a subgoal model proposes a reachable target. Training/representation models make the policy better before execution. Thus, different checks can be necessary for predictions with some of the same properties.

These functions have different latency and reliability requirements. A training-time world model can be slow. A test-time action filter must operate before its decision becomes too late. A planner can tolerate approximation if it replans frequently. A safety monitor must use a conservative approach to uncertainty.

Thus, I recommend this deployment question at the start: will the world model run during control, planning, training or evaluation?

The answer changes the architecture.

It also changes the metrics:

Training-time model: does the downstream policy use fewer real trials, generalize better or recover better?
Planner's model: does it rank short action sequences correctly when long rollouts have errors?
Action filter: does uncertainty calibration let the filter reject dangerous futures without too many rejections of actions that help the task?
Evaluator: does it identify failures as collision risk, task mismatch, high uncertainty, predicted slip or unreachable state?
Subgoal generator: can the lower-level policy reach the visual targets?

I recommend identification of the necessary function in the robot system. Imagined training can help decrease data collection. Evaluation and filtering can help with manipulation deployment. Prediction supervision during training can be the applicable function for a VLA team, without rollout during execution.

Different measurements for future quality and action quality

WAM evaluation has two different sets of tests.

The first set of tests is for world-model quality. Does the imagined future stay visually and semantically consistent? Does it keep object identity, contact order, physical plausibility and state that affects actions? Can a different model or inverse dynamics estimator recover an action that can plausibly cause the imagined change?

The second set of tests is for action-policy quality. Does the policy complete tasks, recover from perturbations and generalize to new objects or scenes? Is operation on real devices successful?

No set of tests is sufficient alone. Visual fidelity metrics can give high scores to an attractive but non-executable future. Task success can hide limited future prediction if the policy behaves mostly as a strong reactive controller. For WAMs, the necessary tests connect the two sets. They include perceptual consistency, physical plausibility, action recoverability, task success, latency and real-robot robustness.

I recommend a measurement of the change in reliable robot behavior from future prediction, together with video metrics and task success.

A first experiment and a reading sequence

I recommend the smallest closed loop that you can run and examine as a first experiment.

First, print one robot episode before training.

Open one episode from an RLDS-style dataset, Open X-Embodiment-style dataset or your simulator logs. Print a sequence of five timesteps, without missing timesteps. For each timestep, record:

Observation keys and shapes
Action shape, units and controller convention
Proprioceptive fields
Language instruction or task ID
Reward, success, discount or termination fields
Robot name, camera names, and available calibration or embodiment metadata

Then examine the episode manually:

Look at frame $t$ . Record the scene in words.
Read action $a_t$ in the robot's controller convention.
Compare frame $t+1$ with the command's expected effect under the controller convention.

These simple checks can identify important errors. An incorrect pairing of $a_t$ with the next frame will train the model with incorrect transition alignment. Different storage conventions for gripper commands across robots can cause the model to average incompatible actions. If success labels occur only at episode termination, a one-step predictor possibly receives no information about the transition's importance.

I then recommend this sequence, with one concept at each step:

Examine episodes before training.
Start in a state-based simulator to examine the control loop.
Use a color-image (RGB) simulator. Add perception.
Train an action-conditioned predictor.
Use it as a short-horizon evaluator or planner.
Compare against a small WAM baseline.

Keep the first project easy to examine

I recommend a small first project, in which you can examine each failure.

Use this checklist:

Select one simulated manipulation task with a clear success metric.
Save episodes as ordered transitions, not loose frames.
Include random actions, scripted successes, scripted failures, and recoveries.
Train an action-conditioned predictor on $(o_t, a_t, o_{t+1})$ .
Add language only after the basic transition model operates correctly.
Compare state observations against RGB observations.
Measure one-step error, multi-step rollout drift and downstream task success as different quantities.
Add an uncertainty estimate or ensemble before you make decisions from long rollouts.
Use the model first as an evaluator or short-horizon planner, not as the full robot system.

This project is sufficient for you to find real problems. They include partial observability, compounding error, action alignment, representation selection and the difference between prediction quality and control quality.

Examine training with imagined transitions

For model-based RL as a training loop, start with DreamerV3.

Read it for one question. Can an agent learn compact latent dynamics and train behavior inside imagined trajectories?

Identify these components in the code or paper:

The replay buffer that stores real environment interaction
The encoder that converts observations into latent state
The recurrent dynamics model that predicts latent states for actions
The reward, value and continuation heads
The actor-critic update in imagined rollouts

On a small task, compare real and imagined rollouts. Before benchmark tuning, examine observation error, reward prediction, actor exploitation, horizon and uncertainty.

Examine the effect of short-horizon planning on action

For continuous control and planning, start with TD-MPC2.

TD-MPC2 examines MPC with task-oriented latent dynamics across continuous-control tasks. This approach gives an alternative to video generation.

I recommend this first examination:

Clone or read the implementation for a simple state-observation control task.
Use RGB observations for the same task.
Compare the changes that make the task less easy when perception is in the control loop.
After this comparison, examine the latent dynamics, value and planning code.

The state-versus-RGB comparison can help very much with this study. State observations give the model compact variables closely related to control. With pixels, the model must learn perception and dynamics together. This change illustrates an important robotics problem. World modeling is less easy when sensors do not give the state directly.

Then change from locomotion or DMControl to a manipulation task, such as pick-cube in ManiSkill. Contact, action dimension, sparse rewards and object pose increase the effects of prediction errors. I recommend this small experiment:

Train or evaluate a state-observation model.
Do the experiment again with RGB observations.
Compare rollout error, task success and sensitivity to planning horizon.
Decrease the planning horizon. Replan more frequently.
Record action-ranking accuracy when the model’s long rollout has visual or numerical errors.

The action ranking is the control result for the test.

Examine the actor, learner and robot timing

DayDreamer is my recommendation for study of the change from a simulator to a real environment.

I do not recommend first-day reproduction without the necessary hardware and laboratory setup. But use the paper as a system-design reading exercise. It shows an organization for Dreamer-style world-model learning on real robots. The components include an actor process, learner process, replay buffer, resets, asynchronous data collection and online adaptation.

The primary question concerns continued learning during robot interaction, not immediate execution of an xArm command. Read for the actor/learner separation, replay writes, reset handling and human intervention. Compare proprioceptive observations with visual observations. Identify the parts that are dangerous without supervision. Many world-model papers do not include this system information, but this information is necessary for real robot learning.

Compare direct policy learning with prediction supervision

For WAMs, first study the VLA baseline.

OpenVLA is not a world model. It is a vision-language-action policy. Its inputs are observation and language, and its output is action. This policy gives a direct comparison. Study the observation-to-action policy before tests of joint future prediction:

o_t, \ell \rightarrow a_t

Open X-Embodiment and RLDS-style datasets supply data for this observation-to-action mapping. They organize cross-robot data as episodes, timesteps, observations, actions, language instructions and metadata. This organization gives the WAM learning problem a clear data structure. Stable learning of the WAM objective is possible only with aligned observation, language, action and next observation.

For a new reader, I recommend this sequence:

Examine RLDS or Open X-Embodiment episodes. Identify the observation, action, language, reward and metadata fields.
Read or run a VLA fine-tuning recipe to examine the direct policy baseline.
Make a small action-conditioned dynamics model for a simulator task.
Compare action-only, future-only, and joint future-plus-action objectives.

A 7B model is not necessary for the small WAM project. Use ManiSkill, Meta-World, RLBench or a different available simulator. Collect random, scripted, successful and failed trajectories. Train three small baselines:

Behavior cloning / VLA-style baseline: predict action from observation and instruction.
World-model baseline: predict next latent state or next observation from observation and action.
Small WAM baseline: predict future state and future action together.

Then measure changes in action generalization, recovery from off-distribution states and evaluation of candidate actions from future prediction. Without better results, the joint objective, representation, data mixture or task selection does not help at this time.

For the WAM comparison, add a controlled inference ablation. Keep future-prediction supervision fixed during training. Compare a direct action path with a path that uses an imagined future. Match the backbone, data and action horizon as fully as possible with that implementation. Report total latency.

This experiment examines the source of better results: training or imagination before action.

A source map after the design questions

After examination of the design alternatives, the sequence of publication can help you study the source papers. Each adds a different input or output:

World Models: this method compresses pixels, predicts latent futures and trains a controller through the learned internal world.
PlaNet and DreamerV3: these methods learn recurrent latent dynamics from pixels, then plan or train behavior in imagined latent rollouts.
MuZero and TD-MPC2: these methods model quantities necessary for decisions. Pixel reconstruction for human inspection is not necessary.
I-JEPA and V-JEPA: these methods predict representations that can help, without a requirement to model each visual detail.
Genie, UniSim, and Cosmos: these methods scale world models for interactive environments, data engines and foundation simulators.
Robot world models and WAMs: these methods examine robot control, uncertainty, rollout drift, and joint modeling of future state and future action.

World Models has different components for vision, memory and control. PlaNet and Dreamer examine latent memory and imagination. MuZero and TD-MPC2 use predictions for decisions. JEPA's benefit changes with its targets. Genie, UniSim and Cosmos add scale/interactivity. WAMs examine the effect of coupled future learning and action learning on control.

Open research problems

WAM research has these open problems:

Architectural coupling: compare cascaded and joint designs with matched scale, data and latency.
Multimodal physical state: add force, tactile, audio, depth, deformation or proprioception inputs when RGB does not contain the decisive signal.
Data mixture design: measure when robot data, simulation, portable human demonstrations and egocentric video give better or worse results together.
Long-horizon drift: prevent compounding prediction errors and action errors across subtasks.
Inference efficiency: make future-aware policies sufficiently fast for closed-loop control.
Safe deployment: before hardware operation, do tests of the causal connection between imagined futures and executable actions.

Video models, robot foundation models and model-based RL have sufficiently related methods for shared research. But the robotics constraints stay the same. A robot executes commands through a body, with latency, partial observations and uncertain contacts. Its physical results cannot be edited after execution.

I recommend one direct test. Does the model help the embodied agent execute actions better on the deployment task, with the applicable latency and safety constraints? If not, the model possibly helps with prediction or representation learning. But its value as a robotics world model is not proven.

Key Takeaways

A video generator helps as a robotics world model only when its predictions connect to executable action, planning, evaluation or safety decisions.
The primary interface is action-conditioned prediction: $o_t, a_t \rightarrow o_{t+1}$ .
Latent models, pixel models, JEPA-style predictors and interactive video models have different tradeoffs between fidelity, efficiency and their value for control.
I recommend control responses to uncertainty when rollout drift can cause physical risk. The design changes for training, planning, filtering or deployment monitoring.
World Action Models couple future world prediction and future action prediction. This coupling connects video priors and robot policies.
Cascaded WAMs are easier to examine. Joint WAMs couple future prediction and action prediction more tightly, but are less easy to optimize.
WAMs have two necessary evaluation axes: future quality and action quality, not video realism alone.
The architecture changes with deployment: training-time imagination, test-time planning, action filtering, and verification have different latency and reliability requirements.

Citation

Use this citation:

Ji, Wenbo. “Embodied AI World Models”. fusheng-ji.github.io (June 2026). https://fusheng-ji.github.io/blog/posts/world-models-for-embodied-ai/

Or use the BibTeX entry:

@article{ji2026worldmodelsforembodiedai,
  title = {Embodied AI World Models},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {June},
  url = {https://fusheng-ji.github.io/blog/posts/world-models-for-embodied-ai/}
}

References

I used ChatGPT (GPT-5.4 Thinking) to write and change this blog. My reading notes on embodied AI world models and robot learning papers supplied the material. Captions identify the sources of official paper figures.

World Models Robotics

Updated 08 November 2026