A person can turn a pen between two fingers. A robot can copy the visible motion and still drop it. A choice of information is necessary for learning from video when the body changes: motion, contact, or the object’s outcome.

Abundant video, missing actions

A person can record an hour of drawer opening, towel folding, valve turning and tool operation. The recording can also show desk rearrangement and recovery from failure. The video contains far more behavioral variety than most robot datasets. But it has almost none of the controller’s inputs.

These inputs include calibrated joint positions, motor commands, measured contact forces, and robot camera images. Usually, the video also has no record of the motion that caused an object transition.

More video cannot supply that mapping by itself. Each method must determine how a visible human event constrains a different body. The body has different joints, sensing, reach, compliance, and contact geometry.

For example, a person can turn a pen between the index and middle fingers. The pixels show that the pen rotated. Optical flow gives a dense description of image motion. A hand tracker may recover an approximate pose.

These signals do not determine which robot links should touch the pen. They do not determine when pressure should transfer between them. They do not determine which joint command will keep the pen from falling. The most visually faithful imitation can be physically useless. A robot motion that looks different from the human motion may cause the correct contact sequence and succeed.

I think of these methods as different choices of interface: the representation supplied from a human demonstration to robot learning. Each method selects information that is easier to extract from the demonstration and use on the robot. The question is what survives the change of body.

This article examines five choices:

• DreamDojo uses continuous latent actions as a scalable control proxy. It learns interaction dynamics from unlabeled video.
• HumanEgo represents hands and objects with interaction-centric tokens. It learns a robot policy directly from minutes of task video.
• Video2Sim2Real reconstructs a simulator-ready digital twin. It uses object-centric keyframes to repair a retargeted skill.
• TopoRetarget maps motion-capture trajectories to dexterous robot hands. It keeps local hand-object contact topology during this mapping.
• EgoEngine translates the two sides of a demonstration. It generates synchronized robot-view observations and executable robot actions.

These systems use different inputs and outputs at different learning stages. Their tasks, hardware, data, metrics, and deployment assumptions are also different. The comparison covers the information each keeps, the machinery necessary for it, and the failures the next stage must handle.

Two earlier research directions give context for these methods. Physical Intelligence's human-to-robot transfer study showed that egocentric human demonstrations can improve broadly pretrained robot policies. EgoMimic studied egocentric video as a scalable imitation-learning source. The next question for these five methods is: When a method uses human video as data, what exactly should be transferred?

What counts as an interface

An interface can be a learned code, relational tokens, a reconstructed simulator, an interaction graph, or a synthetic observation-action pair. It specifies which information robot learning can get from the human demonstration.

Manipulation learning already has names for these requirements. For a skill, a state that lets it run, an expected effect and a way to notice when contact goes wrong are necessary. Kroemer, Niekum, and Konidaris organize these ideas in their review of robot learning for manipulation. They use object-centric state, physical interaction modes, skill preconditions and effects, and execution monitoring. Human video makes each one less easy because the true state and action are usually hidden.

A compact abstraction is:

Here, $h_{1:T}$ is the human demonstration, and $z_{1:T}$ is the chosen interface. The robot-consumable supervision is $y^r_{1:T}$. It can be an action condition, policy target, keyframe reference, trajectory, or paired training example.

The interesting design choice sits in $z$. If it preserves only image similarity, the robot may inherit a pretty but uncontrollable signal. If it preserves too much human-specific geometry, transfer becomes brittle. If it compresses too aggressively, task-critical contact or object state disappears.

Three gaps occur across the five methods.

The visual gap is the difference between the camera view in the human recording and the camera view at robot deployment. Human arms occlude objects differently. Camera height, optics, and motion differ. A robot body introduced at deployment may occupy pixels that never appeared in the human training set. A policy can overfit to the demonstrator’s skin, sleeves, viewpoint, or background even if the task is unchanged.

The action gap is the difference between visible motion and executable command. A video records positions after forces act. Commands are necessary for a controller before the outcome occurs. Human and robot joints are not in one-to-one correspondence. Even perfect human hand pose does not tell us whether a robot should issue a joint target, Cartesian delta, torque, or residual correction.

The dynamics and contact gap is the difference between matching motion and reproducing physical effect. Mesh errors, friction, compliance, actuator lag, object mass, and contact timing can turn a plausible reference into failure. This gap stays after perception has improved and after a kinematic retargeting solution exists.

The three gaps interact. A correction of one does not correct the others. Removal of a human arm from an image addresses a visual-domain problem. It does not make a retargeted action executable.

Reconstruction of a perfect object mesh improves simulation. It does not select the representation a policy should use as input. Learning an action latent can expose controllable dynamics at scale. It does not by itself say what the target robot’s commands mean.

This article compares each interface along six dimensions:

1. Input signal: ordinary RGB, RGB-D, egocentric sensor data, or tracked hand-object motion
2. Preserved invariant: the information claimed to survive the change of body
3. Robot data requirement: whether target-robot trajectories are needed, and where
4. Output form: world-model condition, policy action, refined trajectory, or paired dataset
5. Deployment position: pretraining, direct policy learning, skill acquisition, reference generation, or data-engine generation
6. Primary failure: the error that the interface cannot resolve and therefore sends to the next stage

Preservation is selective and task-dependent. A digital twin can keep every visual detail yet miss the contact transition. Sparse keyframes may work better if they keep the object outcome the robot must reproduce.

Interface I — Continuous latent actions in DreamDojo

DreamDojo starts with scale: a mixture with 44,711 hours of egocentric human interaction video, described in the paper as roughly 44k hours. Most clips lack robot actions, so they lack the input a conventional action-conditioned world model expects.

The obvious alternatives are both unsatisfying. Action-free video prediction lets the model learn that scenes change, but weakens the association between an intervention and its effect. Estimation of human pose supplies labels, but the label space is tied to a human body and to the reliability of a pose pipeline. DreamDojo instead learns a continuous latent action from visual transitions and uses that latent as a unified proxy condition across heterogeneous video.

The latent-action model has an information bottleneck. An encoder infers a compact code from frame pairs. A dynamics decoder uses the initial observation and that code to predict the later observation. The paper uses a 32-dimensional continuous latent. The intended result of the bottleneck is a code that is not a general image container. To be useful, it should represent the change that helps explain the transition: reaching, pulling, rotating, moving an object, or another interaction pattern.

The pretraining objective is:

This does not mean $z_t$ has recovered the true human motor command. It means the code is predictive of an observed transition under a compression constraint. What survives the change of body is therefore best described as controllable interaction dynamics. This is a compact direction in representation space that shows changes in the visible world. It does not commit to human joint coordinates.

The latent makes unlabeled video usable for action-conditioned world-model pretraining. A mapping to robot controls is still necessary.

Target-robot data is still necessary

After pretraining, DreamDojo post-trains on target-robot data. DreamDojo resets the action-conditioning layer so the model can learn the new embodiment’s actual action space and keep interaction knowledge in the broader video model. DreamDojo grounds the proxy at this stage. Human video supplies breadth. The smaller robot dataset teaches what each robot command means.

Robot data is therefore still necessary after pretraining on 44k hours of human video. The model can use target data for adaptation of pretrained dynamics to the robot. Training on both the visual world and its controls from scratch is not necessary.

The system also distills an autoregressive student that generates successive frames from its history. Large video diffusion models are too slow for live teleoperation or online planning when each candidate future takes many denoising steps. The paper reports a distilled student that generates 640×480 frames at 10.81 FPS. It presents long-horizon autoregressive generation, policy evaluation, live teleoperation, and model-based planning as downstream tasks.

What the interface buys

The latent can be extracted automatically from large video pools with different human embodiments and no robot controller logs. Its continuous values can represent graded changes and adapt to high-dimensional robot control more naturally than a small discrete codebook.

DreamDojo also separates the body-specific part early. The world model first learns a broad conditional prior over visual interaction. Only later does it learn what a particular robot action means. For world-model policy evaluation or planning, this is attractive: the deployed policy can query a simulator-like predictor with candidate robot actions and examine possible outcomes.

What is still unresolved

A latent that predicts the next frame can explain that an object moved and still miss the contact configuration that made the motion robust. Visual generation can also smooth over rare or nuanced failure modes. DreamDojo’s paper explicitly notes a limitation in the generation of subtle failures: predicted policy success can be higher than real success. This matters for planning, because a world model that is optimistic exactly where contact becomes delicate may rank the wrong action.

Many latent codes can explain the same visual transition, so the learned axes may not map to stable, human-readable actions. That is acceptable when the model uses the latent internally for pretraining. It becomes less easy if one expects direct manipulation of the latent across environments or fine-grained guarantees about what it controls.

DreamDojo uses broad video to learn how interactions change scenes, then target-robot data to learn the controls. This is applicable to large unlabeled collections when direct deployment from video alone is not necessary.

Interface II — Interaction-centric tokens in HumanEgo

HumanEgo examines almost the opposite regime. It does not start with internet-scale pretraining and later adapt to a robot. Its research question is whether minutes of task-specific egocentric video can train a deployable manipulation policy without task-specific robot demonstrations.

That goal changes what the interface must do. A latent action useful for world-model pretraining can stay semantically loose. For a direct policy trained from a small dataset, a precise state representation is necessary to support closed-loop action. That representation must be stable enough to survive a change of viewpoint and embodiment.

HumanEgo’s representation contains entities and their relations. It starts with recordings from Project Aria glasses, Meta’s egocentric recording device. It uses 3D hand keypoints from tracking, object detection and segmentation, 2D tracking, multi-view triangulation, and orientation estimates. It also uses camera poses from simultaneous localization and mapping (SLAM).

HumanEgo uses the thumb-index pair as a virtual parallel-jaw gripper. Their midpoint gives a human end-effector position, and a hand frame supplies orientation. Fingertip distance becomes a grasp variable.

The visual bridge is partly explicit. HumanEgo can inpaint human-arm regions. Compact keypoint renderings show geometry without a requirement for identification of a particular skin texture or sleeve. More importantly, the policy does not receive hands and objects as independent absolute states. HumanEgo constructs interaction-centric tokens (ICTs).

For each entity, an ICT contains an entity type and its pose in a shared reference frame. It contains the left- and right-hand poses expressed in that entity’s local frame, and a grasp state. In compact form:

The relative transforms are the decisive part. “The right hand is near the cup handle and aligned with its axis” transfers better than “the right hand is at this global coordinate.” The entity frame moves with the object, so approaching, grasping, carrying, and releasing appear as structured changes in hand-object relations. The token set can also change with the number of objects.

Why relative hand–object poses help with little data

It is possible to give a policy a list of object poses and a list of hand poses and let attention discover the relationships. With abundant, clean data, that might be sufficient. HumanEgo has a low-data design. The representation makes the task’s coordinate relations explicit, which is valuable in this regime.

Absolute coordinates put task state, camera placement, table origin and the demonstrator’s reach in one representation. With separate tokens, the policy must infer interaction from a small number of examples. Object-relative poses state it directly: “the hand occupies this pose relative to the cup.”

The policy combines the ICT set with a color (RGB) observation. Conditional flow matching generates a sequence of two-hand poses with six degrees of freedom (6-DoF), plus grasp commands. Flow matching learns a velocity field that changes Gaussian noise into action trajectories. The model can thus represent multiple plausible actions instead of an immediate collapse to one regression target.

HumanEgo adds three auxiliary objectives for prediction to the shared encoder:

• The objective for future object motion in 6-DoF directs the representation toward physical consequences.
• The objective for future entity traces in 2D keeps the representation grounded in visual evolution.
• The objective for future ICT state trains the latent state to model changes in interaction relations.

Each short demonstration supplies training targets for actions, object outcomes, image-space motion and changes in relations. These extra tasks help constrain the learned representation when data is scarce.

What the reported numbers mean

The HumanEgo paper reports 92.5% average success across four real-world tasks with 30 minutes of human video for each task, and 75% with 15 minutes. It also reports a 41 percentage-point improvement over ACT (Action Chunking with Transformers), a robot-imitation baseline, trained with matched-time robot teleoperation in its evaluation. The scope is the paper’s four tasks, the WidowX robot arm setup, perception stack, collection protocol, and trial definition.

The more durable result is the data interface, which uses a small set of structured entity relations. These relations can supply more transferable task state than raw hand and object observations treated independently.

The cost of explicit structure

HumanEgo moves burden from robot demonstration collection into perception. Accurate Aria hand tracking, SLAM, detection, segmentation, point tracking, triangulation and object orientation are necessary for the pipeline. It also uses kinematic latching when an object becomes occluded after grasp. The tracked hand then determines the latched object’s pose. Errors can therefore compound before the policy sees a token.

This is the mirror image of DreamDojo. DreamDojo accepts a scalable learned latent whose meaning is not clear. HumanEgo builds an interpretable relational state whose quality depends on a cascade of estimators. The latter is attractive when task video is short and deployment is direct. It is less applicable to deformables, severe occlusion or ambiguous object identity. It is also less applicable to interactions that rigid 6-DoF entities and a binary grasp cannot summarize.

HumanEgo’s hypothesis is that these relative poses supply enough state for zero-shot policy transfer. The state shows who is acting on what, and in which pose.

Interface III — Digital twins and keyframe anchors in Video2Sim2Real

Video2Sim2Real narrows the input further: a single human manipulation video with color and depth (RGB-D). From that one demonstration, it attempts full-stack dexterous skill acquisition, including reconstruction, retargeting, simulation refinement, sim-to-real correction, and optional spatial generalization.

With one example, statistical learning cannot average away noisy motion estimates. The system instead converts the video into a simulator-ready problem. Off-the-shelf perception modules identify the task and objects. They reconstruct geometry and Unified Robot Description Format (URDF) assets, estimate tabletop poses, and track object motion. They recover a human hand trajectory and retarget it into an initial robot trajectory. Together these make a digital twin: a simulation for tests of candidate robot motion against object geometry and task outcome.

Naive replay of the retargeted trajectory would put too much faith in human pose estimation. A small wrist error at grasp time can move a fingertip to the wrong side of an object. Video2Sim2Real therefore treats the estimated hand trajectory as a prior, not a command. It searches the object motion for three event anchors:

• Contact, when sustained motion of the manipulated object begins
• Interaction, when the manipulated object changes relative to the target object
• Detachment, when the method infers release after contact

The exact heuristics use tracked object motion and the task type. Their conceptual role is more general: these are the moments when the robot’s configuration must be correct for the demonstrated object transition to stay possible.

Repair the moments that determine the outcome

At the contact keyframe, the method uses reconstructed object geometry and task semantics to optimize the hand-object relation. For grasping, it can generate and filter grasp candidates in simulation. For pushing or pulling, it uses object-flow direction and a surface contact point for fingertip alignment. At the interaction keyframe, the method estimates the desired manipulated-object pose from 3D object flow. It combines that pose with the corrected hand-object transform. At detachment, the method adjusts the hand pose so release occurs at the desired object pose.

The corrected robot configurations become anchors. Inverse kinematics maps them to joint space, and interpolation repairs the raw trajectory between them. This design assumes that the object outcome at causal interaction events is more trustworthy than every frame of the human hand path.

Video2Sim2Real starts from hand imitation, but it does not stop there. Simulation can change a motion that does not have the demonstrated effect on the object.

The method separates global perception error from local contact-dynamics error during simulation-to-real (sim-to-real) transfer. An imitation-learning (IL) module maps a noisy object point cloud to corrected keyframe hand poses. The system queries it before the robot enters the workspace to prevent robot-induced occlusion. A residual reinforcement-learning (RL) controller learns local finger-level corrections under physics randomization. The base trajectory gives the geometric reference. The residual makes corrections for friction, contact, and other local dynamics.

The decomposition matters. A residual policy for every correction can destroy the useful smoothness and collision properties of the refined trajectory. Conversely, for supervised imitation of contact physics, generation of successful labels across many randomized conditions is necessary. The two-stage correction gives each learner a narrower job.

An optional module for collision-aware motion planning can connect the learned local skill to novel object arrangements. That extends the single demonstration beyond exact replay: perception selects a local target pose, planning reaches it safely, and the refined dexterous segment handles interaction.

What the digital twin makes testable

The digital twin changes the type of supervision available. Before reconstruction, the system has a noisy observation of human motion.

After reconstruction, it can do counterfactual tests. Does this robot grasp close around the object? Does the object follow? Is the release pose correct? Is the path collision-free? Simulation makes those questions testable.

But that power is conditional on asset quality. Usable geometry, pose, scale, articulation, object identity and task interpretation are necessary for the twin. A polished mesh with the wrong collision geometry is worse than a crude correct one. The rigid digital-twin assumptions used here may not be applicable to a task with deformable material, fluid or hidden articulation.

The three keyframes are intentionally sparse and heuristic. They are applicable when contact, interaction and detachment transitions are sufficient for the task description. They are less expressive for continuous in-hand reorientation, rolling contact, or manipulation where contact switches repeatedly without a clear detachment. The paper also identifies limitations around complex in-hand rotation and the absence of tactile feedback.

Video2Sim2Real’s bet is that one demonstration can be enough if the system recovers a simulator in which object-centric constraints can repair the motion. It preserves what happened to the object at the moments that made the skill succeed.

Interface IV — Contact topology in TopoRetarget

TopoRetarget begins with hand-object trajectories from a motion-capture glove and object geometry, rather than ordinary RGB video. It largely removes perception error for this comparison of transfer between bodies.

If the embodiment gap were mainly an estimation problem, high-quality tracked hand motion should make transfer straightforward. TopoRetarget shows that it does not. Even with dense hand keypoints and object pose, a robot hand can match human pose and destroy the contacts that made the motion work. This hand has different finger lengths, joint axes, link geometry, and reachable set.

Pen spinning makes the issue concrete. Imagine that every robot fingertip is as close as possible to its corresponding human fingertip. The resulting pose can look correct in a side-by-side visualization.

But a millimeter-scale change may place the pen on the wrong side of a link, remove a supporting contact, or introduce interpenetration. It may also prevent the intended transfer from one finger to another. Pose error is continuous. Contact success can change abruptly.

TopoRetarget therefore preserves local topology of hand-object interaction rather than pose alone.

From bone directions to an interaction mesh

The method begins with a bone-direction warm-up. Rather than match human and robot joint angles, which are not directly comparable, it aligns relative directions of adjacent finger bones in wrist-centered frames. A temporal smoothness term discourages frame-to-frame jumps. This gives an initial robot configuration that roughly follows the demonstrated hand articulation and uses the robot’s kinematic variables.

The second stage adds the object. At each frame, TopoRetarget combines 21 hand keypoints with samples from the object surface. It builds a sparse interaction graph with Delaunay tetrahedralization (division of a point set into non-overlapping tetrahedra) on the source human configuration. It then uses the same graph connectivity again for the robot configuration. The method now compares human and robot under the same local neighborhood structure.

For each graph vertex, a distance-weighted Laplacian coordinate gives its relation to nearby vertices. Close hand-object edges receive more weight than distant ones. Optimization has a target: the robot graph reproduces the source graph’s Laplacian coordinates. The optimization also has bone-direction priors and temporal regularity, plus constraints on joint feasibility and penetration.

The topology language can sound more abstract than it is. Suppose one side of the pen locally supports a human finger segment, while another finger approaches from the opposite side. The graph encodes those neighborhoods. The robot is free to use a different absolute pose if the local interaction structure stays similar and feasible.

This sparse geometric proxy directs the optimizer toward the coupled hand-object configuration that pose-only objectives ignore.

Retargeting quality changes policy learnability

The output is a robot reference trajectory, not a policy. TopoRetarget uses a relatively minimal Proximal Policy Optimization (PPO) tracking setup to learn residual actions around those references. It uses object, hand-link, joint, and smoothness objectives plus dynamics randomization. This separation lets the paper’s authors do a test of an important hypothesis. Better references can make the downstream reinforcement-learning problem easier even when the authors keep the policy learner simple.

The authors report the best contact precision and alignment among their evaluated baselines on the ContactPose hand–object dataset. They report a 40.6 percentage-point improvement in pen-spinning training success over existing baseline references. They report zero-shot transfer demonstrations on the Wuji dexterous hand for cube reorientation and pen spinning.

The deeper lesson is that representation quality appears downstream as optimization difficulty. A reference with the wrong contact structure rewards reinforcement learning for tracking a bad reference. It also makes a physically different strategy necessary for the learner. A contact-aware reference gives RL a corridor that already contains viable behavior.

What motion-capture input teaches video methods

TopoRetarget does not solve human-video perception. It assumes tracked hand-object motion and an object mesh. Another interface would be necessary to recover those from casual RGB video. It can preserve only the interactions that its keypoints, surface samples, graph and signed-distance constraints can express. The representation leaves out soft contact, tactile force, and deformable objects.

Yet the method is diagnostic for every video pipeline in this article. Better pose estimation does not make human and robot embodiments the same. Even if a future foundation model reconstructs every human finger perfectly, a contact-feasible mapping is still necessary for the robot.

TopoRetarget’s bet is that local contact neighborhoods are a more transferable description of dexterous intent than human pose similarity. It preserves which parts of hand and object support the interaction, and how those local relations evolve.

Interface V — Robotized demonstrations in EgoEngine

EgoEngine produces paired robot demonstrations: video with the human replaced by the target robot, synchronized with an executable robot action trajectory.

For this, it is necessary to cross the visual and action gaps together. If only the action is retargeted, the downstream visuomotor policy still trains on images of human arms and encounters robot arms at deployment. If only the video is edited to look robotic, the generated frames have no trustworthy action labels. EgoEngine’s interface is the alignment between the two.

The system first reconstructs an object-centric digital twin from egocentric recordings. It combines camera geometry, depth, human and object masks, hand poses, an object mesh, and a temporally consistent 6-DoF object trajectory. This common state supplies input for two parallel branches.

The visual branch removes the human arms and hands with segmentation masks and inpainting. It renders the robot along the optimized trajectory from the egocentric camera. A two-pass rendering procedure estimates which robot pixels should be visible given object occlusion, then blends the render into the inpainted real scene. The result keeps the background, lighting and task object from the source video and adds the target embodiment.

The action branch first applies human-centric inverse-kinematic retargeting under joint-limit and self-collision constraints. This gives a robot-shaped motion prior, but not necessarily an executable skill. The branch then optimizes against the demonstrated object trajectory in simulation.

Object motion is the common clock

The action optimizer compares the object trajectory produced in simulation with the object trajectory observed in the human video. A thresholded object-pose error determines whether a rollout stays viable. Rather than train a strong RL controller over the entire long horizon, EgoEngine divides the trajectory into chunks and escalates solver strength as needed.

It first replays the retargeted reference. If replay fails the object-centric criterion, model-predictive control (MPC) supplies local correction. If that is still not sufficient, residual reinforcement learning supplies stronger adaptation. The paper calls this a Monte Carlo Tree Search (MCTS)-style mode-switching strategy. It is a lightweight heuristic search over solver modes, not full MCTS.

RL on every frame would be expensive and could change working segments unnecessarily. Replay keeps those segments close to the human motion. Stronger solvers concentrate simulation effort on chunks that are not easy.

Once both branches finish, EgoEngine pairs the synthesized robot observation $\tilde{o}_t$ with the optimized robot action $\tilde{a}_t$. A downstream visuomotor policy is trained on the resulting synthetic dataset:

The paper reports, to the authors’ knowledge, the first zero-shot visuomotor policy for dexterous manipulation learned from egocentric human videos without real-robot demonstrations. A target robot model, simulation, digital-twin reconstruction and substantial optimization are still necessary for the engine. “Zero-shot” refers to downstream real-robot deployment without real-robot demonstration data for policy training.

A robot-looking video is not a robot demonstration

This distinction is the core of EgoEngine. Generative editing can make an image appear to contain a robot, but causal alignment is necessary for a policy dataset. The observation and label do not agree if the rendered hand is three centimeters away from the action trajectory. They also do not agree if a generated object moves independently of the simulated contact. Training on more such data can make the policy worse.

EgoEngine avoids unconstrained video generation for the robot itself. EgoEngine renders the robot from the optimized trajectory and composites it with occlusion awareness. The digital twin and the observed object path give the visual and action branches a common geometry and timeline. What survives transfer is the agreement between robot-view pixels, actions, and the object’s progress.

This interface is broader than action retargeting and more expensive. Errors can contaminate both sides of supervision:

• Mask: a poor human mask leaves visual artifacts.
• Inpainting: it may invent object texture.
• Twin: depth or object-pose error corrupts the digital twin.
• Retargeting: it may initialize the wrong contact.
• Simulation: it may accept an unrealistic trajectory.
• Blending: it can make a domain gap.

The downstream policy can learn any of these artifacts.

The paper gives visual quality and digital-twin reconstruction as bottlenecks and shows that perception failures cascade. Pinch grasps are especially unforgiving because small pose or geometry errors change contact. Unlike a latent interface, which can hide uncertainty in representation space, a generated demonstration commits to specific pixels and specific actions. Its labels look complete even when they are wrong.

EgoEngine’s bet is that a costly, synchronized conversion can make the exact modality a standard robot policy expects. It preserves the agreement between what the robot would see and what the robot should do.

Where the five interfaces sit

The five methods change different learning stages:

• DreamDojo changes world-model pretraining. Human video becomes action-conditioned prediction with a latent proxy. DreamDojo grounds target-robot actions later.
• HumanEgo changes direct policy learning. Its relational tokens and flow-matching head give robot actions from task demonstrations without task-specific robot data.
• Video2Sim2Real changes skill acquisition. A digital twin and object keyframes turn one video into a corrected, deployable skill.
• TopoRetarget changes the trajectory reference supplied to RL.
• EgoEngine changes data generation. It supplies the paired modality used to train a conventional visuomotor policy.

The table uses the same vocabulary for all five methods. “Target-robot action data” means demonstrations or action-labeled target-robot trajectories used to learn the method, not the existence of a robot model, URDF, simulator, or deployment platform.

This comparison also exposes why “uses no robot data” is too coarse. HumanEgo can learn a policy without task-specific robot demonstrations. The target robot’s kinematic definition and a control interface are still necessary. Video2Sim2Real and EgoEngine avoid real demonstration collection at the cost of simulator construction and optimization. TopoRetarget avoids robot demonstrations but assumes unusually strong human motion capture. DreamDojo uses target-robot data and human video at a scale the other interfaces do not attempt.

There is no free embodiment transfer. The cost moves between labels, perception, geometry, simulation, optimization, and deployment data.

Choosing an interface

Select according to the supervision your data supports and the output necessary for your system.

If you have a large unlabeled video pool

Use a latent-action interface when breadth is the asset and action annotation is the bottleneck.

The goal is to make passive video conditionable enough for world-model pretraining, then ground the control space with smaller target-robot data. This is the DreamDojo regime. It is especially relevant for downstream prediction, planning, policy evaluation or a dynamics prior reusable across many tasks.

Do not select it if the immediate deliverable is a guaranteed executable trajectory from one demonstration. A latent world model amortizes knowledge across a distribution. It does not replace task-level contact repair.

If you have minutes of task video and a policy is necessary

Use an interaction-centric state when perception can recover relative poses for a manageable set of rigid scene entities.

This is the HumanEgo regime. The representation puts task geometry into the coordinate system. Each short trajectory can supply training data for several future-prediction targets.

The approach becomes less attractive when object tracking is unreliable or force, rather than pose, determines the task. It is also less attractive when rigid transforms cannot give a representation of the entities. In those cases, a concise relational token may not include exactly what matters.

If you have one RGB-D demonstration and a workable simulator

Use a digital twin and object-centric keyframes to recover one skill, not to learn statistically from many examples.

This is the Video2Sim2Real regime. Reconstruction converts observation into a testbed. Keyframes give correction targets at the transitions that determine the object outcome. IL and residual RL handle different parts of sim-to-real error.

The requirement goes beyond “a simulator is available.” The manipulated objects and interaction must be reconstructable at useful fidelity. If construction of the twin is less easy than collection of robot demonstrations, the interface loses its economic advantage.

If you have high-quality hand-object motion for a dexterous hand

Use contact-aware retargeting when source tracking is strong and the bottleneck is the mapping between hand morphologies.

It keeps local contact topology. This is the TopoRetarget regime. The output is a better reference for downstream RL, not a perception-to-action policy.

This interface is particularly valuable for in-hand skills where pose similarity is misleading and reference quality dominates RL learnability. It is less relevant for casual untracked video unless paired with a robust hand-object reconstruction stage.

If more robot-view, action-labeled training data is necessary

Use joint visual/action translation when the downstream learner expects ordinary robot observation-action pairs.

This is the EgoEngine regime, for an organization willing to keep a digital-twin and rendering pipeline in working condition. Its advantage is compatibility: the generated data can train a standard visuomotor policy. Its risk is silent label corruption on both sides of the pair.

A good data engine therefore must have rejection criteria, confidence estimates and consistency checks, not only more generated samples. It should be able to say that a human video could not be converted reliably.

Combining the five interfaces

These interfaces solve different pressures, so the most promising path may be compositional.

A large latent-action world model could supply broad priors about plausible interactions and object transitions. Interaction-centric tokens could give a task policy a compact closed-loop state, especially under viewpoint change. A digital twin could do tests of candidate motions and correct sparse object-outcome anchors. Contact topology could refine the final dexterous reference before reinforcement learning. A rendering branch could then supply aligned robot-view observations for policy distillation.

Clear handoffs would be necessary for their combination:

1. Latent dynamics: which interventions plausibly change the scene?
2. Interaction state: which entities are coupled, and how does their relation change?
3. Outcome anchors: which object transitions define progress?
4. Contact-aware execution: which robot configuration can realize those transitions?
5. Observation-action alignment: what will the deployed policy see when it executes them?

Such a stack would also make uncertainty easier to find. A world-model error is different from an object-pose error. A retargeting infeasibility is different from a rendering artifact. An opaque end-to-end model makes those failures not easy to diagnose and expensive to correct.

Follow one cup through the proposed stack

In this example, a person lifts a cup from a table and puts it on a tray.

A latent-action world model could suggest plausible cup transitions, but its change code is not a target robot command. Grounding in that robot's action space would first be necessary for the proposal. An interaction-centric state could then express the wrist relative to the cup and the cup relative to the tray. The two stages can agree on the desired object outcome without a shared latent vocabulary.

An adapter would have to recover the cup pose, object identity, coordinate frame and timestamp from the prediction. Only then could it use the prediction as a geometric target. If those estimates are uncertain, the prediction should stay a proposal.

A digital twin could do a test of whether a candidate grasp lifts the cup and whether the release leaves it on the tray. Sparse anchors could constrain those events and let the intervening hand path change. For a dexterous hand, local contact structure might also be necessary to keep the cup supported. That structure cannot be recovered from a wrist pose and a binary grasp flag alone.

The original hand keypoints and object surface would have to survive the earlier compression. Otherwise, they would have to be estimated again. Different action adapters are therefore necessary for HumanEgo's virtual parallel-jaw gripper and TopoRetarget's articulated hand references. One is not a finer version of the other.

Only after a candidate robot trajectory passes the object-outcome checks should a rendering branch supply its observation labels. It must render the optimized robot motion with the same camera calibration, occlusion order, and timeline used by the action branch. If the human video stayed unchanged, the pair would break. It would also break with labels that pair generated cup motion with unrelated robot commands.

Each handoff must preserve object identity, units, coordinate frames, timing, and uncertainty. After an adapter changes the motion, the system must do another check of the physical outcome. DreamDojo's failure-prediction limits and EgoEngine's contact-modeling limits make that last check necessary.

There is a tension here. Explicit interfaces improve debuggability but can cap performance at the quality of their estimators. Learned representations absorb ambiguity but are less easy to verify. Both may be necessary for a practical system: learned priors and geometric constraints. The priors are for cases where scale and ambiguity dominate. The constraints are for cases where contact and safety make explicit checks necessary.

Conclusion

Human video cheaply records behavior. Conversion of human video into robot supervision makes a choice necessary. That choice determines what to keep when the camera, body, joints and dynamics change.

DreamDojo preserves a compact description of interaction dynamics. HumanEgo preserves hand-object relations. Video2Sim2Real preserves object outcomes at causal keyframes. TopoRetarget preserves local contact structure. EgoEngine preserves alignment between robotized observations and executable actions. Each method removes some information and keeps other information.

More video expands the observed behaviors. To turn them into action, the representation must keep intervention, contact, and outcome: the parts that constrain what the robot can do next.

Companion research instrument

I made Human2Robot around a question that continued to bother me. A demonstration shows what happened, but what must a robot understand to make it occur again?

The study uses one simple transfer. First the robot copies the visible motion. Then it grounds the path in the object and goal. Finally it monitors contact and the expected object effect. The scene and disturbances stay fixed, so each failure points back to a missing part of the skill description rather than to a different task.

This is a deterministic MuJoCo representation ablation.

@article{ji2026fromhumanvideotorobotaction,
  title = {From Human Video to Robot Action: Five Interfaces Across the Embodiment Gap},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/from-human-video-to-robot-action/}
}

References

1. Gao, Shenyuan, William Liang, Kaiyuan Zheng, et al. “DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos.” 2026. Project page.
2. Wang, Zhi (Leo), Botao He, Kelin Yu, et al. “HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos.” 2026. Project page.
3. Han, Yunhai, Jianuo Qiu, Linhao Bai, et al. “Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video.” 2026. Project page.
4. Wu, Jielin, Shenzhe Yao, Guanqi He, et al. “TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation.” 2026. Project page.
5. Liu, Yangcen, Shuo Cheng, Xinchen Yin, et al. “EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations.” 2026. Project page.
6. Physical Intelligence. “Emergence of Human to Robot Transfer in VLAs.” 2025.
7. Ma, Yecheng Jason, et al. “EgoMimic: Scaling Imitation Learning through Egocentric Video.” 2024.
8. Kroemer, Oliver, Scott Niekum, and George Konidaris. “A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms.” Journal of Machine Learning Research, 2021.

Updated 09 November 2026