From Human Video to Robot Action: Five Interfaces Across the Embodiment Gap

There is no shortage of video showing people doing things. There is a shortage of trajectories that a robot can execute. Between those two facts lies the embodiment gap: the quiet set of decisions that determines whether a human demonstration becomes a useful control signal or remains only an interesting movie.

Abundant video, missing actions

A person can record an hour of manipulation without touching a robot. They can open drawers, fold towels, turn valves, use tools, rearrange a desk, and improvise around failure. The resulting video contains far more behavioral variety than most robot datasets. It also contains almost none of the variables a robot controller expects. There are no calibrated joint positions, no motor commands, no measured contact forces, no robot camera image, and usually no explicit record of which motion caused which object transition.

This mismatch is easy to understate. “Learn from human video” sounds like a data-scaling claim: replace expensive robot demonstrations with cheap human recordings, then let a sufficiently large model absorb the difference. But video volume does not remove the missing interface. If the training target is robot action, every method must decide how a visible human event should constrain a different body with different kinematics, sensing, reach, compliance, and contact geometry.

Suppose a person turns a pen between the index and middle fingers. The pixels tell us that the pen rotated. Optical flow gives a dense description of image motion. A hand tracker may recover an approximate pose. None of those signals determines which robot links should touch the pen, when pressure should transfer between them, or which joint command will keep the pen from falling. The most visually faithful imitation can be physically useless. Conversely, a robot motion that looks unlike the human motion may produce the right contact sequence and succeed.

The recent human-to-robot literature is therefore more legible when treated as a debate about interfaces, not as one race toward a universal method. Each system selects an intermediate representation that is easier to obtain from a human demonstration and easier for a robot to consume. That selection is a bet about what can survive a change of body.

This article examines five such bets:

• DreamDojo treats continuous latent actions as a scalable control proxy for learning interaction dynamics from unlabeled video.
• HumanEgo turns hands and objects into interaction-centric tokens and learns a robot policy directly from minutes of task video.
• Video2Sim2Real reconstructs a simulator-ready digital twin and uses object-centric keyframes to repair a retargeted skill.
• TopoRetarget preserves local hand-object contact topology while mapping motion-capture trajectories to dexterous robot hands.
• EgoEngine translates both sides of a demonstration, generating synchronized robot-view observations and executable robot actions.

These systems do not consume the same input, produce the same output, or enter the robot-learning stack at the same point. Their evaluations also differ in tasks, hardware, data, metrics, and deployment assumptions. The useful comparison is not which number is largest. It is which invariant each interface protects, what machinery it needs to do so, and which failure it pushes downstream.

Two earlier lines of work help place the moment. Human2Robot showed how web-scale human activity can participate in general robot-policy training, while EgoMimic studied egocentric video as a scalable imitation-learning source. The five methods here sharpen the next question. Once human video is admitted as data, what exactly should be transferred?

What counts as an interface

I use interface to mean the intermediate contract between a human observation or demonstration and robot-consumable supervision. It is not necessarily an API, a single tensor, or one neural module. It may be a learned latent, a set of relational tokens, a reconstructed simulator, a geometric interaction graph, or a synthetic observation-action pair. What matters is that it defines what the robot side is allowed to trust.

This contract has an older and more concrete manipulation-learning vocabulary. A skill needs a state in which it can be executed, an effect it is expected to produce, and a way to notice when contact moves the system into the wrong mode. Kroemer, Niekum, and Konidaris organize these ideas as object-centric state, physical interaction modes, skill preconditions and effects, and execution monitoring in their review of robot learning for manipulation. Human video makes each one harder because the true state and action are usually hidden.

A compact abstraction is:

where $h_{1:T}$ is the human demonstration, $z_{1:T}$ is the chosen interface, and $y^r_{1:T}$ is robot-consumable supervision: an action condition, policy target, keyframe reference, trajectory, or paired training example. The interesting design choice sits in $z$. If it preserves only image similarity, the robot may inherit a pretty but uncontrollable signal. If it preserves too much human-specific geometry, transfer becomes brittle. If it compresses too aggressively, task-critical contact or object state disappears.

Three gaps recur across the five methods.

The visual gap is the difference between what the human camera observes and what the robot camera will observe. Human arms occlude objects differently. Camera height, optics, and motion differ. A robot body introduced at deployment may occupy pixels that never appeared in the human training set. A policy can overfit to the demonstrator’s skin, sleeves, viewpoint, or background even if the task is unchanged.

The action gap is the difference between visible motion and executable command. A video records positions after forces have acted; a controller needs commands before the outcome occurs. Human and robot joints are not in one-to-one correspondence. Even perfect human hand pose does not tell us whether a robot should issue a joint target, Cartesian delta, torque, or residual correction.

The dynamics and contact gap is the difference between matching motion and reproducing physical effect. Mesh errors, friction, compliance, actuator lag, object mass, and contact timing can turn a plausible reference into failure. This gap remains after perception has improved and after a kinematic retargeting solution exists.

The boundaries interact, but they should not be collapsed. Removing a human arm from an image addresses a visual-domain problem; it does not make a retargeted action executable. Reconstructing a perfect object mesh improves simulation; it does not decide which representation a policy should condition on. Learning an action latent can expose controllable dynamics at scale; it does not by itself provide the target robot’s command semantics.

I will compare each interface along six dimensions:

1. Input signal: ordinary RGB, RGB-D, egocentric sensor data, or tracked hand-object motion.
2. Preserved invariant: the information claimed to survive the change of body.
3. Robot data requirement: whether target-robot trajectories are needed, and where.
4. Output form: world-model condition, policy action, refined trajectory, or paired dataset.
5. Deployment position: pretraining, direct policy learning, skill acquisition, reference generation, or data-engine generation.
6. Primary failure: the error that the interface cannot resolve and therefore sends to the next stage.

That framing makes one distinction especially important: invariance is not information preservation in general. It is selective preservation under a task. A digital twin that reconstructs every visual detail but misses the manipulated object’s contact transition is a poor interface. A sparse keyframe representation that discards most frames may be excellent if those frames anchor the object outcome the robot must reproduce.

Interface I — Continuous latent actions in DreamDojo

DreamDojo starts from the largest-scale version of the problem. Its central resource is not a handful of task demonstrations but a mixture containing 44,711 hours of egocentric human interaction video, described in the paper as roughly 44k hours. Most of those videos do not come with robot actions. A conventional action-conditioned world model cannot simply ingest them, because the condition it expects is absent.

The obvious alternatives are both unsatisfying. Treating the video as action-free prediction lets the model learn that scenes change, but weakens the association between an intervention and its effect. Estimating human pose supplies labels, but the label space is tied to a human body and to the reliability of a pose pipeline. DreamDojo instead learns a continuous latent action from visual transitions and uses that latent as a unified proxy condition across heterogeneous video.

The latent-action model is built around an information bottleneck. An encoder infers a compact code from frame pairs; a dynamics decoder uses the initial observation and that code to predict the later observation. The paper uses a 32-dimensional continuous latent. The bottleneck is meant to prevent the code from becoming a general image container. To be useful, it should represent the change that helps explain the transition: reaching, pulling, rotating, moving an object, or another interaction pattern.

A simplified view of the pretraining objective is:

This does not mean $z_t$ has recovered the true human motor command. It means the code is predictive of an observed transition under a compression constraint. The preserved invariant is therefore best described as controllable interaction dynamics: a compact direction in representation space that says how the visible world is changing, without committing to human joint coordinates.

That distinction is the reason DreamDojo belongs in this comparison. Latent action is not a magical translation from pixels to torques. It is an interface that makes unlabeled interaction video look action-conditioned enough to pretrain a world model.

The target robot still has to enter

After pretraining, DreamDojo post-trains on target-robot data. The action-conditioning layer is reset so the model can learn the new embodiment’s actual action space while retaining interaction knowledge in the broader video model. This is where the proxy is grounded. Human video supplies breadth; the smaller robot dataset supplies command semantics.

This division of labor is easy to miss if one reads “44k hours” as a claim that robot data has disappeared. It has not. DreamDojo changes what robot data must teach. Instead of teaching both the visual world and the robot interface from scratch, target data can focus on adapting a pretrained dynamics model to the embodiment.

The system also includes an autoregressive distillation pipeline. Large video diffusion models are too slow for live teleoperation or online planning if each candidate future takes many denoising steps. The paper reports a distilled student generating 640×480 frames at 10.81 FPS and presents long-horizon autoregressive generation, policy evaluation, live teleoperation, and model-based planning as downstream uses. Those are within-paper results under DreamDojo’s own model and hardware conditions, not a speed comparison against the other four interfaces.

What the interface buys

The latent interface has an unusually favorable scaling property. It can extract proxy actions automatically from large video pools with inconsistent human embodiments and no robot controller logs. Because the action is continuous rather than a small discrete codebook, it can represent graded changes and adapt to high-dimensional robot control more naturally than a handful of symbolic commands.

It also places the embodiment boundary early. The world model first learns a broad conditional prior over visual interaction. Only later does it learn what a particular robot action means. For applications such as policy evaluation or model-based planning, this is attractive: the deployed policy can query a simulator-like predictor with candidate robot actions and inspect possible outcomes.

What remains unresolved

Predictive sufficiency is not physical sufficiency. A latent can explain that an object moved without identifying the contact configuration that made the motion robust. Visual generation can also smooth over rare or nuanced failure modes. DreamDojo’s paper explicitly notes a limitation in generating subtle failures: predicted policy success can be higher than real success. This matters for planning, because a world model that is optimistic exactly where contact becomes delicate may rank the wrong action.

The interface also inherits an identifiability problem. Many latent codes can explain the same visual transition, and the learned axes need not correspond to stable, human-readable actions. That is acceptable when the latent is used internally for pretraining. It becomes harder if one expects direct manipulation of the latent across environments or fine-grained guarantees about what it controls.

DreamDojo’s bet is therefore clear: preserve enough of the intervention-to-outcome structure to learn a broad robot world model, then spend target-robot data to name the controls. It is the right interface when unlabeled scale is the scarce opportunity and direct deployment from video alone is not the requirement.

Interface II — Interaction-centric tokens in HumanEgo

HumanEgo attacks almost the opposite regime. It does not begin with internet-scale pretraining and later adapt to a robot. It asks whether minutes of task-specific egocentric video can train a deployable manipulation policy without task-specific robot demonstrations.

That goal changes what the interface must do. A latent action useful for world-model pretraining can remain semantically loose. A direct policy trained from a small dataset needs a state representation precise enough to support closed-loop action and stable enough to survive a change of viewpoint and embodiment.

HumanEgo’s answer is to represent the scene as entities and their relations. Its perception pipeline begins with Project Aria recordings. It uses tracked 3D human hand keypoints, object detection and segmentation, 2D tracking, multi-view triangulation, orientation estimation, and SLAM-derived camera poses. The thumb-index pair is treated as a virtual parallel-jaw gripper: their midpoint gives a human end-effector position, a hand frame supplies orientation, and fingertip distance becomes a grasp variable.

The visual bridge is partly explicit. Human-arm regions can be inpainted, while compact keypoint renderings expose geometry without forcing a policy to identify a particular skin texture or sleeve. More importantly, the policy does not receive hands and objects as independent absolute states. HumanEgo constructs interaction-centric tokens (ICTs).

For each entity, an ICT contains an entity type, its pose in a shared reference frame, the left- and right-hand poses expressed in that entity’s local frame, and a grasp state. In simplified form:

The relative transforms are the decisive part. “The right hand is near the cup handle and aligned with its axis” transfers better than “the right hand is at this global coordinate.” The entity frame moves with the object, so approaching, grasping, carrying, and releasing appear as structured changes in hand-object relations. The token set can also vary with the number of objects.

Why independent object and hand state is not enough

It is possible to give a policy a list of object poses and a list of hand poses and let attention discover the relationships. With abundant, clean data, that might work. HumanEgo is designed for the low-data regime, where forcing the right coordinate system into the representation is valuable.

Absolute coordinates entangle task state with camera placement, table origin, and the demonstrator’s reach. Separate tokens also make interaction an inference the policy must relearn from few examples. Object-relative hand poses state the relation directly. The invariant is not “the cup is here” or “the hand is here,” but “the hand occupies this pose relative to the cup.” That is much closer to what determines manipulation progress.

The policy combines the ICT set with an RGB observation and generates a horizon of bimanual 6-DoF poses and grasp commands using conditional flow matching. Flow matching learns a velocity field that transports Gaussian noise toward action trajectories, giving the model a way to represent multiple plausible actions rather than collapsing immediately to one regression target.

HumanEgo adds three auxiliary prediction objectives to the shared encoder:

• future 6-DoF object motion, which pushes the representation toward physical consequence;
• future 2D entity traces, which keeps the representation grounded in visual evolution;
• future ICT state, which asks the latent state to model how interaction relations change.

These heads do not merely add labels. They make each short demonstration supervise action, object outcome, image-space motion, and relational dynamics at once. In a small-data setting, that is a practical form of representation regularization.

What the reported numbers mean

The HumanEgo paper reports 92.5% average success across four real-world tasks using 30 minutes of human video per task, and 75% with 15 minutes. It also reports a 41 percentage-point improvement over ACT trained with matched-time robot teleoperation in its evaluation. These are striking results, but their correct scope is the paper’s four tasks, WidowX setup, perception stack, data-collection protocol, and trial definition. They should not be compared numerically with DreamDojo’s world-model metrics or with dexterous-hand results in later sections.

The more durable result is the data interface: a small set of structured entity relations can carry more transferable task state than raw hand and object observations treated independently.

The cost of explicit structure

HumanEgo moves burden from robot demonstration collection into perception. The pipeline depends on accurate Aria hand tracking, SLAM, detection, segmentation, point tracking, triangulation, and object orientation. It also uses kinematic latching when an object becomes occluded after grasp, tying object pose to the tracked hand. Errors can therefore compound before the policy sees a token.

This is the mirror image of DreamDojo. DreamDojo accepts a learned latent that is hard to interpret but scalable. HumanEgo builds an interpretable relational state whose quality depends on a cascade of estimators. The latter is attractive when task video is short and deployment is direct; it is less comfortable when the scene contains deformables, severe occlusion, ambiguous object identity, or interactions that cannot be summarized by rigid 6-DoF entities and a binary grasp.

HumanEgo’s bet is that interaction geometry, expressed in the right coordinate system, is the minimum sufficient state for zero-shot policy transfer. It preserves who is acting on what, and in which relative pose.

Interface III — Digital twins and keyframe anchors in Video2Sim2Real

Video2Sim2Real narrows the input further: a single RGB-D human manipulation video. From that one demonstration, it attempts full-stack dexterous skill acquisition, including reconstruction, retargeting, simulation refinement, sim-to-real correction, and optional spatial generalization.

With one example, statistical learning cannot average away noisy motion estimates. The system instead converts the video into a simulator-ready problem. Off-the-shelf perception modules identify the task and objects, reconstruct object geometry and URDF assets, estimate tabletop poses, track object motion, recover a human hand trajectory, and retarget it into an initial robot trajectory. Together these components form a digital twin: not a cinematic replica of the scene, but a simulation state in which a candidate robot motion can be tested against object geometry and task outcome.

Naively replaying the retargeted trajectory would put too much faith in human pose estimation. A small wrist error at grasp time can move a fingertip to the wrong side of an object. Video2Sim2Real therefore treats the estimated hand trajectory as a prior, not a command. It searches the object motion for three event anchors:

• contact, when sustained motion of the manipulated object begins;
• interaction, when the manipulated object changes relative to the target object;
• detachment, when release is inferred after contact.

The exact heuristics depend on tracked object motion and the task type. Their conceptual role is more general: these are the moments when the robot’s configuration must be correct for the demonstrated object transition to remain possible.

Repair the moments that determine the outcome

At the contact keyframe, the method uses reconstructed object geometry and task semantics to optimize the hand-object relation. For grasping, it can generate and filter grasp candidates in simulation. For pushing or pulling, it uses object-flow direction and a surface contact point to align the fingertip appropriately. At the interaction keyframe, the desired manipulated-object pose is estimated from 3D object flow, then combined with the corrected hand-object transform. At detachment, the hand pose is adjusted so release occurs at the desired object pose.

The corrected robot configurations become anchors. Inverse kinematics maps them to joint space, and interpolation repairs the raw trajectory between them. This design makes a strong statement about the invariant: the object outcome at causal interaction events is more trustworthy than every frame of the human hand path.

It is a useful reversal. Most retargeting pipelines begin by asking how closely the robot can imitate the human hand. Video2Sim2Real begins with that estimate but gives the simulator authority to revise it when it does not produce the demonstrated object effect.

The method also distinguishes global perception error from local contact-dynamics error during sim-to-real transfer. An imitation-learning module maps a noisy object point cloud to corrected keyframe hand poses. It is queried before the robot enters the workspace, avoiding robot-induced occlusion. A residual reinforcement-learning controller then applies local finger-level corrections under physics randomization. The base trajectory handles geometry; the residual handles friction, contact, and other local dynamics.

This decomposition matters because asking one residual policy to fix everything can destroy the useful smoothness and collision properties of the refined trajectory. Conversely, asking supervised imitation to cover contact physics requires generating successful labels across many randomized conditions. The two-stage correction gives each learner a narrower job.

An optional collision-aware motion-planning module can connect the learned local skill to novel object arrangements. That extends the single demonstration beyond exact replay: perception chooses a local target pose, planning reaches it safely, and the refined dexterous segment handles interaction.

Why the digital twin is an interface, not just infrastructure

The digital twin changes the type of supervision available. Before reconstruction, the system has a noisy observation of human motion. After reconstruction, it can ask counterfactual questions: does this robot grasp close around the object, does the object follow, is the release pose correct, and is the path collision-free? Simulation turns passive evidence into a testable constraint.

But that power is conditional on asset quality. The twin needs usable geometry, pose, scale, articulation, object identity, and task interpretation. A polished mesh with the wrong collision geometry is worse than a crude correct one. A task involving deformable material, fluid, or hidden articulation may not admit the rigid digital-twin assumptions used here.

The three keyframes are intentionally sparse and heuristic. They work when the task can be described by contact, interaction, and detachment transitions. They are less expressive for continuous in-hand reorientation, rolling contact, or manipulation where contact switches repeatedly without a clear detachment. The paper also identifies limitations around complex in-hand rotation and the absence of tactile feedback.

Video2Sim2Real’s bet is that one demonstration can be enough if the system recovers a simulator in which object-centric constraints can repair the motion. It preserves what happened to the object at the moments that made the skill succeed.

Interface IV — Contact topology in TopoRetarget

TopoRetarget is the boundary case in this article. Its input is not ordinary RGB video. It begins with human hand-object trajectories captured using a motion-capture glove, together with object geometry. The perception problem is therefore largely removed. That is exactly why it belongs here.

If the embodiment gap were mainly an estimation problem, high-quality tracked hand motion should make transfer straightforward. TopoRetarget shows that it does not. Even with dense hand keypoints and object pose, a robot hand with different finger lengths, joint axes, link geometry, and reachable set can match human pose while destroying the contacts that made the motion work.

Pen spinning makes the issue concrete. Imagine aligning every robot fingertip as closely as possible to the corresponding human fingertip. The resulting pose can look correct in a side-by-side visualization. But a millimeter-scale change may place the pen on the wrong side of a link, remove a supporting contact, introduce interpenetration, or prevent the intended transfer from one finger to another. Pose error is continuous; contact success can change abruptly.

TopoRetarget therefore preserves local hand-object interaction topology rather than pose alone.

From bone directions to an interaction mesh

The method begins with a bone-direction warm-up. Rather than match human and robot joint angles, which are not directly comparable, it aligns relative directions of adjacent finger bones in wrist-centered frames. A temporal smoothness term discourages frame-to-frame jumps. This produces an initial robot configuration that roughly follows the demonstrated hand articulation while respecting the robot’s kinematic variables.

The second stage adds the object. At each frame, TopoRetarget combines 21 hand keypoints with samples from the object surface. It builds a sparse interaction graph through Delaunay tetrahedralization on the source human configuration, then reuses the same graph connectivity for the robot configuration. Human and robot are now compared under the same local neighborhood structure.

For each graph vertex, a distance-weighted Laplacian coordinate describes its relation to nearby vertices. Close hand-object edges receive more weight than distant ones. Optimization asks the robot graph to reproduce the source graph’s Laplacian coordinates while also respecting bone-direction priors, temporal regularity, joint feasibility, and penetration constraints.

The topology language can sound more abstract than it is. If a human finger segment is locally supported by one side of the pen while another finger approaches from the opposite side, the graph encodes those neighborhoods. The robot is free to use a different absolute pose as long as the local interaction structure remains similar and feasible.

This is not perfect topology in the mathematical sense of contact modes under all perturbations. It is a sparse, geometry-aware proxy. But it directs the optimizer toward what pose-only objectives ignore: the configuration of hand and object as one coupled system.

Retargeting quality changes policy learnability

The output is a robot reference trajectory, not a policy. TopoRetarget uses a relatively minimal PPO tracking setup to learn residual actions around those references, with object, hand-link, joint, and smoothness objectives plus dynamics randomization. This separation lets the paper test an important hypothesis: better references can make the downstream reinforcement-learning problem easier even when the policy learner is held simple.

The authors report the best contact precision and alignment among their evaluated baselines on ContactPose, a 40.6 percentage-point improvement in pen-spinning training success over existing baseline references, and zero-shot transfer demonstrations on the Wuji Hand for cube reorientation and pen spinning. Again, those numbers describe TopoRetarget’s evaluation and should not be folded into a leaderboard with policy-success rates from different platforms.

The deeper lesson is that representation quality appears downstream as optimization difficulty. A reference with the wrong contact structure forces reinforcement learning to discover a physically different strategy while being rewarded for tracking the bad reference. A contact-aware reference gives RL a corridor that already contains viable behavior.

What the mocap boundary case teaches video methods

TopoRetarget does not solve human-video perception. It assumes tracked hand-object motion and an object mesh; obtaining those from casual RGB video would require another interface. It can also only preserve interactions that its keypoints, surface samples, graph, and signed-distance constraints can express. Soft contact, tactile force, and deformable objects are outside that contract.

Yet the method is diagnostic for every video pipeline in this article. Better pose estimation does not make embodiment disappear. Even if a future foundation model reconstructs every human finger perfectly, the robot still needs a contact-feasible mapping. The perception ceiling reveals a control floor.

TopoRetarget’s bet is that local contact neighborhoods are a more transferable description of dexterous intent than human pose similarity. It preserves which parts of hand and object support the interaction, and how those local relations evolve.

Interface V — Robotized demonstrations in EgoEngine

EgoEngine takes the most literal interpretation of “turn human video into robot data.” Its output is not only a state representation or a motion reference. It produces paired robot demonstrations: a video in which the human has been replaced by the target robot, synchronized with an executable robot action trajectory.

This requires crossing the visual and action gaps together. If only the action is retargeted, the downstream visuomotor policy still trains on images of human arms and encounters robot arms at deployment. If only the video is edited to look robotic, the generated frames have no trustworthy action labels. EgoEngine’s interface is the alignment between the two.

The system first reconstructs an object-centric digital twin from egocentric recordings. It combines camera geometry, depth, human and object masks, hand poses, an object mesh, and a temporally consistent 6-DoF object trajectory. From this common state, two branches proceed in parallel.

The visual branch removes the human arms and hands using segmentation masks and inpainting. It renders the robot along the optimized trajectory from the egocentric camera. A two-pass rendering procedure estimates which robot pixels should be visible given object occlusion, then blends the render into the inpainted real scene. The result preserves the background, lighting, and task object from the source video while introducing the target embodiment.

The action branch first applies human-centric inverse-kinematic retargeting under joint-limit and self-collision constraints. This yields a robot-shaped motion prior, but not necessarily an executable skill. The branch then optimizes against the demonstrated object trajectory in simulation.

Object motion is the common clock

The action optimizer compares the object trajectory produced in simulation with the object trajectory observed in the human video. A thresholded object-pose error determines whether a rollout remains viable. Rather than train a strong RL controller over the entire long horizon, EgoEngine divides the trajectory into chunks and escalates solver strength as needed.

It first tries replay of the retargeted reference. If replay fails the object-centric criterion, it uses model-predictive control for local correction. If that remains insufficient, residual reinforcement learning supplies stronger adaptation. The paper describes this as an MCTS-style mode-switching strategy, while being explicit that it is a lightweight heuristic search over solver modes rather than full Monte Carlo Tree Search.

This adaptive schedule is important for data generation. Applying RL to every frame would be expensive and could unnecessarily alter segments that already work. Retaining replay where possible preserves the human motion prior; escalating only difficult chunks concentrates simulation effort around embodiment failures.

Once both branches finish, EgoEngine pairs the synthesized robot observation $\tilde{o}_t$ with the optimized robot action $\tilde{a}_t$. A downstream visuomotor policy is trained on the resulting synthetic dataset:

The paper reports, to the authors’ knowledge, the first zero-shot visuomotor dexterous policy learned from egocentric human videos without real-robot demonstrations. The exact claim is narrower than “robot learning without robots”: a target robot model, simulation, digital-twin reconstruction, and substantial optimization are still part of the engine. “Zero-shot” refers to downstream real-robot deployment without real-robot demonstration data for policy training.

A robot-looking video is not a robot demonstration

This distinction is the core of EgoEngine. Generative editing can make an image appear to contain a robot, but a policy dataset needs causal alignment. If the rendered hand is three centimeters away from the action trajectory, or if a generated object moves independently of the simulated contact, the observation and label disagree. Training on more such data can make the policy worse.

EgoEngine avoids unconstrained video generation for the robot itself. The robot is rendered from the optimized trajectory and composited with occlusion awareness. The digital twin and the observed object path tie the visual and action branches to a common geometry and timeline. The preserved invariant is therefore view-action alignment under object-centric task progress.

This interface is broader than action retargeting and more expensive. Errors can contaminate both sides of supervision. A poor human mask leaves visual artifacts; inpainting may invent object texture; depth or object-pose error corrupts the twin; retargeting may initialize the wrong contact; simulation may accept an unrealistic trajectory; blending can create a domain gap; and the downstream policy may learn any of those artifacts.

The paper identifies visual quality and digital-twin reconstruction as bottlenecks and shows that perception failures cascade. Pinch grasps are especially unforgiving because small pose or geometry errors change contact. Unlike a latent interface, which can hide uncertainty in representation space, a generated demonstration commits to specific pixels and specific actions. Its labels look complete even when they are wrong.

EgoEngine’s bet is that a costly, synchronized conversion can create the exact modality a standard robot policy expects. It preserves the agreement between what the robot would see and what the robot should do.

Where the five interfaces sit

The five methods are sometimes discussed together because they all begin from human behavior and aim toward robot competence. Their interfaces, however, enter at different stages.

DreamDojo changes world-model pretraining. Human video becomes action-conditioned prediction through a latent proxy; target-robot actions are grounded later. HumanEgo changes direct policy learning. Its relational tokens and flow-matching head produce robot actions from task demonstrations without task-specific robot data. Video2Sim2Real changes skill acquisition. A digital twin and object keyframes turn one video into a corrected, deployable skill. TopoRetarget changes the trajectory reference supplied to RL. EgoEngine changes data generation, creating the paired modality used to train a conventional visuomotor policy.

The table below uses one vocabulary across all five. “Robot data” means demonstrations or action-labeled target-robot trajectories used to learn the method, not the existence of a robot model, URDF, simulator, or deployment platform.

This comparison also exposes why “uses no robot data” is too coarse. HumanEgo can avoid task-specific robot demonstrations but still needs the target robot’s kinematic definition and a control interface. Video2Sim2Real and EgoEngine avoid real demonstration collection by spending simulator construction and optimization. TopoRetarget avoids robot demonstrations but assumes unusually strong human motion capture. DreamDojo uses target-robot data, yet leverages human video at a scale the other interfaces do not attempt.

There is no free embodiment transfer. The cost moves between labels, perception, geometry, simulation, optimization, and deployment data.

Choosing an interface

The right question is not “which paper won?” It is “which intermediate supervision can my data support, and what output does my system need?”

If you have a large unlabeled video pool

Use a latent-action interface when breadth is the asset and action annotation is the bottleneck. The goal is to make passive video conditionable enough for world-model pretraining, then ground the control space with smaller target-robot data. This is the DreamDojo regime. It is especially relevant when the downstream use is prediction, planning, policy evaluation, or a reusable dynamics prior across many tasks.

Do not choose it if the immediate deliverable is a guaranteed executable trajectory from one demonstration. A latent world model amortizes knowledge across a distribution; it does not replace task-level contact repair.

If you have minutes of task video and need a policy

Use an interaction-centric state when the scene can be represented as a manageable set of rigid entities and the perception pipeline can recover their relative poses. This is the HumanEgo regime. The representation bakes task geometry into the coordinate system and allows every short trajectory to supervise several future-prediction targets.

The approach becomes less attractive when object tracking is unreliable, the task depends on force rather than pose, or the entities cannot be described by rigid transforms. In those cases, a concise relational token may omit exactly what matters.

If you have one RGB-D demonstration and a workable simulator

Use a digital twin and object-centric keyframes when a single skill must be recovered rather than statistically learned from many examples. This is the Video2Sim2Real regime. Reconstruction converts observation into a testbed; keyframes focus correction on the transitions that determine the object outcome; IL and residual RL handle different parts of sim-to-real error.

The hidden requirement is not merely “a simulator is available.” The manipulated objects and interaction must be reconstructable at useful fidelity. If building the twin is harder than collecting robot demonstrations, the interface has lost its economic advantage.

If you have high-quality hand-object motion for a dexterous hand

Use contact-topology-preserving retargeting when source tracking is strong and the bottleneck is the mapping between hand morphologies. This is the TopoRetarget regime. The output is a better reference for downstream RL, not a perception-to-action policy.

This interface is particularly valuable for in-hand skills where pose similarity is misleading and reference quality dominates RL learnability. It is less relevant for casual untracked video unless paired with a robust hand-object reconstruction stage.

If you need more robot-view, action-labeled training data

Use joint visual/action translation when the downstream learner expects ordinary robot observation-action pairs and the organization is willing to maintain a digital-twin and rendering pipeline. This is the EgoEngine regime. Its advantage is compatibility: the generated data can train a standard visuomotor policy. Its risk is silent label corruption on both sides of the pair.

A good data engine therefore needs rejection criteria, confidence estimates, and consistency checks, not only more generated samples. It should be able to say that a human video could not be converted reliably.

The likely hybrid is not a sixth interface

These interfaces solve different pressures, so the most promising path may be compositional.

A large latent-action world model could supply broad priors about what interactions and object transitions are plausible. Interaction-centric tokens could give a task policy a compact closed-loop state, especially under viewpoint change. A digital twin could test candidate motions and correct sparse object-outcome anchors. Contact topology could refine the final dexterous reference before reinforcement learning. A rendering branch could then produce aligned robot-view observations for policy distillation.

The point is not to concatenate every module. Each stage should have a clear contract:

1. Latent dynamics: which interventions plausibly change the scene?
2. Interaction state: which entities are coupled, and how is their relation evolving?
3. Outcome anchors: which object transitions define progress?
4. Contact-aware execution: which robot configuration can realize those transitions?
5. Observation-action alignment: what will the deployed policy see while executing them?

Such a stack would also make uncertainty easier to locate. A world-model error is different from an object-pose error; a retargeting infeasibility is different from a rendering artifact. When one opaque model owns the entire conversion, those failures are difficult to diagnose and expensive to correct.

There is a tension here. Explicit interfaces improve debuggability but can cap performance at the quality of their estimators. Learned representations absorb ambiguity but are harder to verify. A practical system may need both: learned priors where scale and ambiguity dominate, geometric constraints where contact and safety demand accountability.

Conclusion

Human video is not cheap robot data. It is cheap evidence about behavior. Turning that evidence into supervision requires a decision about what should remain true when the camera, body, kinematics, and dynamics change.

DreamDojo preserves a compact description of interaction dynamics. HumanEgo preserves hand-object relations. Video2Sim2Real preserves object outcomes at causal keyframes. TopoRetarget preserves local contact structure. EgoEngine preserves alignment between robotized observations and executable actions. Each gains leverage by discarding information as well as preserving it.

That is the broader research lesson. Dynamic visual modeling becomes useful for embodied intelligence only when it retains intervention, contact, and outcome—the parts of a video that constrain what an agent can do next. More video expands the space of observed behavior. A well-chosen interface turns that space into action.

Companion research instrument

I built Human2Robot around a question that kept bothering me: a demonstration shows what happened, but what must a robot understand to make it happen again?

The study uses one simple transfer. First the robot copies the visible motion. Then it grounds the path in the object and goal. Finally it monitors contact and the expected object effect. The scene and disturbances stay fixed, so each failure points back to a missing part of the skill description rather than to a different task.

This is still a deterministic MuJoCo representation ablation, not a learned policy or a real-world transfer claim. The reviewed source video is also still pending, so the public page currently shows its frozen trace as an explicit prototype fallback. Runtime reports, limitations, and input hashes remain available for readers who want to audit the comparison.

The source repository keeps the broader planning, recovery, and teaching infrastructure inspectable without making those systems the project’s primary claim.

@article{ji2026fromhumanvideotorobotaction,
  title = {From Human Video to Robot Action: Five Interfaces Across the Embodiment Gap},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/from-human-video-to-robot-action/}
}

References

1. Gao, Shenyuan, William Liang, Kaiyuan Zheng, et al. “DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos.” 2026. Project page.
2. Wang, Zhi (Leo), Botao He, Kelin Yu, et al. “HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos.” 2026. Project page.
3. Han, Yunhai, Jianuo Qiu, Linhao Bai, et al. “Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video.” 2026. Project page.
4. Wu, Jielin, Shenzhe Yao, Guanqi He, et al. “TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation.” 2026. Project page.
5. Liu, Yangcen, Shuo Cheng, Xinchen Yin, et al. “EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations.” 2026. Project page.
6. Physical Intelligence. “Human to Robot: Learning Robot Policies from Human Videos.” 2025.
7. Ma, Yecheng Jason, et al. “EgoMimic: Scaling Imitation Learning through Egocentric Video.” 2024.
8. Kroemer, Oliver, Scott Niekum, and George Konidaris. “A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms.” 2020.