A person reaches toward a virtual jar, closes two fingers around the lid, and twists. Does the lid turn? Does the jar stay on the table? Does the next frame arrive in time for the person to adjust their grip?

This article compares 2D pose, SMPL bodies and tracked head and hand motion as controls for video world models. It also examines when each control becomes available. It uses Generated Reality and Directing the World to show two requirements for live motion control. Response latency must be low, and objects must respond and stay changed.

Motion can supply information that text does not give

A prompt names a scene. A first frame gives the reference appearance. Camera paths or point trajectories, as in Tora, may supply spatial intent. The model generates a clip, then the user judges it. That is applicable to composition, but interaction is still largely unspecified.

"Open the jar" says what should eventually happen. It does not say how the hand approaches, where fingers close or how the wrist rotates. It also does not say when the motion should stop. A keyboard can supply a few direction signals. A game controller adds continuous axes. A designer must still map its indirect signal vocabulary to the scene.

Human motion already contains the spatial structure the experience is trying to express. Head orientation says where attention moves. A wrist pose says where a hand intends to act. A full-body trajectory says how an actor crosses space, balances, turns, and coordinates with other actors.

Motion gives continuous spatial input at the speed the person acts. That makes it attractive for interactive video world models, which generate the next frames of a scene in response to input. A tracked skeleton that is rendered as diffusion guidance only supplies pose. The world also has to respond.

The distinction is between pose compliance, following a body’s shape, and controlling what that body changes.

A model can reproduce the silhouette of a raised arm while it ignores the object the hand was meant to lift. It can follow a walking skeleton while it lets the floor slide under the subject. It can draw fingers around a cup but fail to preserve which side of the handle they occupy. It can generate a perfectly plausible camera move that cancels the motion the user intended to see. In each case, the condition is visible in the pixels, yet the person does not control the world in a meaningful sense.

The central claim of this article is:

A pose sequence is not yet a control signal. It becomes one when the generated world responds causally, continuously, and with the right consequences.

Recent work makes this distinction easier to see because the field now spans three different temporal regimes. Systems such as Animate Anyone, MimicMotion, and UniAnimate treat a complete pose sequence as an offline animation specification. Directing the World places structured human motion inside a long-horizon autoregressive world-video model and composes it with camera control. Generated Reality goes further: it streams live head and hand tracking into a causal generator and sends the video to a headset.

These systems examine different forms of control. This article examines the motion representation, its model input path, its timing and the consequences it should preserve.

What makes motion an interface

A mouse movement changes a cursor’s position through a defined mapping, coordinate system and delay. For human-motion control, the effect and timing of a tracked movement must also be clear.

Let the tracker observe a human state $h_t$ at time $t$. A representation function $\phi$ converts it into a control $c_t$. A causal generator $G$ advances its internal visual state $s_t$:

The generated frame $x_{t+1}$ supplies visible feedback to the user. After the user sees it, they select the next movement. That closes the feedback loop. This article examines five requirements for the loop.

The motion must be observable

A control estimate must be available when the generator uses the input. An offline, cleaned motion-capture sequence is not the same as live headset tracking with occlusion, lost joints and jitter. Cleaner offline tracking can make a benchmark less representative of live operation.

Sensing is also different at different scales. A camera may recover a torso’s 2D pose but miss fingertips. A headset tracks its own six-degree-of-freedom pose but sees hands only in its field of view. Full-body optical tracking supplies clean geometry at a deployment cost. Each arrangement has its own uncertainty.

The coordinate system must be stable

A 2D wrist location names a pixel. A 3D wrist transform names a pose in camera, body or world coordinates. A Skinned Multi-Person Linear (SMPL) body sequence includes root motion, joint rotations and shape. Camera motion affects these representations differently.

While you keep one hand still, turn your head. The hand moves across the image, but its world pose may stay fixed. Image-space skeletons can make that look like hand motion. Projection into the current view is still necessary for world-space joints. The ambiguities that the model must resolve are different for different coordinate systems.

The condition must be available on time

Offline control can include the entire future. A live tracker supplies the latest state, perhaps predicted slightly ahead to compensate for delay. Training with future pose tokens can give misleadingly strong results when those tokens will be unavailable online.

Include sensing, tracking, buffering, encoding, denoising, decoding, transport and display in the latency calculation. High generation throughput can still feel slow if the generator must complete a large chunk before its first frame appears.

Controls must operate together without interference

Motion shares the model with text, reference appearance, camera movement and previous scene state. A new condition changes how attention is divided among them.

Subject motion and camera motion each cause optical flow. When a dancer moves left and the camera pans right, the model must separate body displacement from viewpoint change. Conditioning modules that operate alone may interfere when they operate together.

The motion must cause a consequence

Consequences beyond character animation are necessary for world control. A pushed button should depress and keep its new state. A wheel should turn about its axis rather than be repainted beneath a palm. Walking behind a table should change occlusion order consistently.

A generated world can stay uncertain without perfect agreement with a hidden simulator. What matters is a coherent relation between the intervention and its outcome. Otherwise motion directs a foreground performance against an unrelated backdrop.

For each method, this article examines the input, the information it keeps, its model input location and its timing. It also examines the controlled scene variables and errors that pass to subsequent components.

What each motion representation preserves

More geometric structure removes some ambiguities and adds dependencies. The following representations have different tradeoffs.

2D skeletons: strong pixels, weak depth

A rendered skeleton or heatmap places joints as points or Gaussian channels and connects bones. Its image coordinates align directly with target pixels. They specify the elbow’s image location.

This spatial alignment is one reason pose-guided animation became practical before general 3D-aware control. ControlNet established a reusable pattern that attaches spatial conditions such as edges, depth, segmentation and human pose to a powerful pretrained diffusion backbone. Animate Anyone combined a pose guider with a dedicated ReferenceNet and temporal layers. This separated the problems of motion compliance, appearance preservation and inter-frame continuity.

A 2D skeleton leaves depth ambiguous. A hand that approaches the camera can resemble one that grows in the image. Crossed fingers can share several 3D interpretations. Off-frame joints disappear, and a camera move changes the projection of a stationary body.

MimicMotion accounts for uncertain detections with confidence-aware guidance and stronger losses in relevant regions. Progressive latent fusion extends output length, but the full driving pose stays an offline input. This improves animation but does not supply a causal sensing/feedback loop.

DensePose-style maps, which assign each body pixel a point on a body surface template, add torso shape and limb extent beyond sparse bones. They still depend on the visible surface and suffer occlusion and camera changes. Clothing, props, hair and hands can move beyond the body template.

3D joints and SMPL: the body in different views

3D joints give information about a kinematic configuration in chosen coordinates. SMPL also represents pose-dependent geometry and identity-dependent shape in a compact animatable body.

Root translation gives information about movement through a scene. Joint rotations give information about articulation. The same motion can be reprojected into another view or assigned to another subject. This separates actor movement from any one image.

The generator must encode, render, project or tokenize 3D parameters to affect images. Incorrect body and camera calibration can produce floating feet or sliding bodies. SMPL represents bodies better than loose garments, held objects or exact hand-contact surfaces. It supplies a body scaffold, but interaction state is incomplete.

A tennis swing gives shoulder, elbow and wrist locations. Separate racket and ball state is necessary to specify contact. Sitting can place the pelvis correctly while the chair shifts or deforms. Better viewpoint handling alone does not add object causality.

Head and hand parameters: interaction at first-person scale

A headset already measures camera pose. A wrist transform plus joint articulation gives a high-rate hand representation without reconstruction of the entire body.

Head pose moves the view and indicates attention. Metric hand articulation is a direct vocabulary for first-person interaction because it distinguishes configurations that overlap in the image.

Small errors matter at contact. Noise harmless for a dancing silhouette can put a fingertip inside a lid. Hands leave view, occlude one another and approach image edges. The generator must render the intended hand and keep its target object.

Datasets show the sensing burden. HOT3D supplies egocentric multi-view streams with tracked 3D hands, objects, and cameras, including UmeTrack hand-tracking and MANO hand-model annotations. GigaHands expands the diversity of bimanual activity with 34 hours, 56 subjects, 417 objects, and 14,000 motion clips, according to its authors. These datasets make it easier to learn the hand-object geometry necessary for such an interface.

Hybrid 2D-3D control: two imperfect views of the same hand

Hybrid control supplies these two inputs: a rendered image-space skeleton and 3D parameters. The skeleton anchors image placement. The parameters clarify depth, wrist rotation and articulation.

The errors complement each other. Correct 3D joints projected with the wrong camera can be misplaced. A correctly placed 2D skeleton can hide finger depth order. Fusion lets the model use each signal for the ambiguity it resolves.

Generated Reality's ablation makes this concrete. The authors compare hand-pose parameter injection, binary masks, skeleton video, and a hybrid of skeleton video plus 3D parameters on HOT3D. In their setup, the hybrid achieves the strongest reported hand-accuracy metrics and keeps competitive video quality. This is evidence for the complementarity of the signals in that model and dataset.

Learned motion features: useful compression, uncertain semantics

Learned features can encode poses, flow or tracked parameters, smooth noise and compress history into a form suited to video latents.

Compression can make control more difficult to interpret. A latent direction that mixes wrist and camera motion is difficult to correct. Video-reconstruction training may keep visible motion without contact state. Supervision must specify what should survive: timing, part correspondence, object response or another measurable behavior.

Image-space controls align easily but leave depth ambiguous. Parametric controls offer stable geometry at an encoding cost. Hybrids use more data and compute to address these two limits. Learned features depend on what their objectives reward.

How motion enters a video model

Motion features must reach spatial layout, temporal changes and denoising. They must not override appearance or camera intent. The way they enter the model matters alongside their representation.

Concatenation and residual control

Concatenation of pose images or latents with noisy video lets the first projection mix spatially aligned values. Extra channels change that input layer, and a strong condition may dominate useful pretrained features.

ControlNet-style branches process conditions separately and add residuals to backbone blocks. At their zero initialization, the projections do not change pretrained outputs. Aligned timing is also necessary for video. Flickering pose features or a mismatched frame schedule still cause errors.

Token addition and concatenation

Concatenation can add motion tokens at the end of the sequence of video tokens for attention to use. It increases sequence length. A learned spatial association is still necessary for a hand parameter at time $t$.

Token addition projects the control into the same shape as video patch tokens and adds the vectors element by element. Its cost is low, and it keeps the token count. Meaningful alignment between control embeddings and video tokens is also necessary. In Generated Reality, the authors evaluate token concatenation, token addition, adaptive layer normalization, and cross-attention for hand-pose parameters. Token addition is the strongest of those parameter-only variants on their hand-accuracy metrics. Their final hybrid uses channel-wise skeleton-video conditioning plus added hand-parameter features.

Cross-attention and adaptive normalization

Cross-attention lets video features query separate motion tokens. It suits non-image-shaped controls or varying entity counts and can be added at selected blocks.

The model must learn the correspondence between joints and regions. This can be difficult on small tracking datasets. Adaptive layer normalization (AdaLN) efficiently scales and shifts activations, but those global changes alone do not locate a fingertip.

Use the alignment already available in the representation. Pixel-aligned skeletons favor early spatial fusion. Cross-attention may be applicable to 3D entities or motion tokens. A compact global mode such as camera style or timestep may use adaptive normalization. Otherwise, the model must learn the known geometry again during training.

Part-aware temporal coherence

PoseAnything addresses part identity under large articulation. Corresponding body regions can move far across pixels. This can confuse temporal attention or cause a loss of fine detail.

The method receives a reference image and an arbitrary skeletal sequence, including non-human skeletons. It encodes reference and pose inputs into latent features and concatenates them by channel. Its Part-aware Temporal Coherence Module segments the skeleton into parts and dilates those segments into masks. It uses attention patterns to match corresponding parts across frames. It then does cross-attention between the matched regions.

The interface is not only "place joints here". It adds "this moving region is the same semantic part as that earlier region."

PoseAnything separates subject and camera guidance with different classifier-free guidance (CFG) anchors. The broader diagnosis matters: those motions overlap in image space. A defined method to combine them is necessary.

Why controls fight

Suppose the subject moves ten pixels right between frames. The same observed flow could come from the subject, a leftward camera pan, a deformation of the body, or a combination. Text may also imply motion: "a runner sprinting" pushes the model toward one distribution even if the supplied skeleton is nearly static. A reference image may resist the large viewpoint change requested by the camera path because the unseen side of the person has no visual evidence.

Stronger pose guidance can damage the subject's appearance or texture. Camera guidance can make the subject slide. Reference attention can keep the clothing but resist large arm and leg movements. Separate encoders alone cannot resolve this. Training must vary the conditions independently enough to teach their roles.

This motivates staged training. Train motion control with static cameras. Then add camera movement. Train hand and camera encoders separately. Then train them jointly. While you adapt the control paths, keep or freeze the expensive video prior.

Explicit controls must coexist with motion the prior already predicts.

When controls become available

For interaction, examine when a control is known, when it can change and when the person sees its effect. Faster generation alone does not answer these timing questions.

Offline animation: the future is already known

Offline pose-guided animation receives a full reference sequence $c_{1:T}$. The model can process all frames jointly or divide them into overlapping windows. Bidirectional temporal attention can use a subsequent pose to resolve an earlier ambiguity. Progressive latent fusion can smooth the boundaries between windows. The output may be a minute long. Because the motion script is fixed before generation starts, no one can change frame 120 in response to frame 119.

This suits film, avatars, fashion, dance and performance transfer. UniAnimate combines reference, pose and noise with iterative first-frame conditioning. Wan-Animate uses aligned skeleton/facial features for animation and replacement. OmniHuman-1 examines scaling with mixed motion conditions. These have value as offline tools.

When an offline generator renders a known sequence quickly, it has high throughput. A live interface must also use input from the next movement without a restart or revision of an already planned clip.

Chunked generation: controls arrive in blocks

A causal or chunk-wise model generates a block from previous frames and its controls. The model does not repeat full-clip denoising because it uses cached history. New controls can arrive at chunk boundaries.

Chunked generation has a control horizon. If the model generates twelve frames at once, controls for those twelve frames must be available before the chunk starts. A late hand movement cannot change frames already in the decoding stage. Smaller chunks decrease that delay but increase overhead and may weaken temporal coherence. Larger chunks amortize computation but make the interface feel more buffered.

A prescribed SMPL script and camera path can control blocks while memory keeps the scene. For live interaction, updates must also be exchanged during rollout.

Live interaction: the person closes the loop

At time $t$, a live system reads recent movement, generates and displays a response, then lets the person select $h_{t+1}$. The video therefore changes the next control input.

That loop has four timing quantities:

• Tracking rate: the frequency of estimates of the body, head or hands
• Control consumption rate: how frequently the generator uses a new condition
• Display rate: how frequently completed frames reach the user
• Response latency: the time from an observed movement to its effect in a visible frame

Frames per second gives only one part of this information. Eleven FPS can occur with 1.4 seconds of response latency if the system waits for a chunk. A model may generate faster than display speed but use new control inputs only at half-second intervals. A tracker may run at headset rate while the generator uses a downsampled trajectory. For interaction, these differences shape agency more than a single throughput number.

People adapt to the system. With response delays, they move their hands more slowly. With tracking uncertainty, they make larger movements. They also stop at unexpected responses. Fixed-dataset pose scores cannot give a full evaluation of this feedback behavior.

The Generated Reality study includes tests of live first-person feedback. The Directing the World study includes tests of long-horizon body/camera composition. Their shared motion input serves different purposes.

Case study I: Generated Reality closes the loop

Generated Reality asks whether video can create a hand-controlled experience without manually built scene assets. Text such as “turn the wheel” leaves hand order, wrist rotation and grip unspecified. Conventional virtual reality (VR) uses scene geometry, rendering and interaction code for those details.

The system uses a Meta Quest 3 to track the user's head and hands. The head supplies a six-degree-of-freedom camera pose. Each hand is represented with UmeTrack parameters: a wrist transformation plus 20 joint angles. The generator receives these two inputs: a 2D projection of the hand skeleton and those 3D parameters.

Consider the same stationary cup seen from two head positions. This example shows the reason for the Plücker-ray camera condition. Each pixel corresponds to a ray through the scene. A camera matrix describes the whole view. A ray map gives the generator a geometric condition at each pixel. With camera center $\mathbf{o}$ and unit ray direction $\mathbf{d}_{u,v}$ in a shared world frame, one convention is

The direction gives the ray’s orientation. The cross-product term gives the line’s location relative to the world origin. This six-channel map changes when the camera moves, even when the cup stays still. It gives the model a method to attribute image displacement to viewpoint rather than hand motion. CameraCtrl's representation study gives an explanation of this pixelwise encoding and compares it with raw camera parameters. The map supplies geometry, not scene depth or contact state, and cannot enforce camera–hand disentanglement on its own.

A variational autoencoder (VAE) calculates latents from video and skeleton frames. The visual branch uses channel concatenation of these latents before patchification. A motion encoder calculates token features from hand parameters. A camera encoder calculates per-frame Plücker embeddings from head poses. The model adds the hand and camera embeddings to the visual tokens before diffusion-transformer blocks.

Why head and hand control have to be learned together

A head turn changes projected hands and background. Hand-only control can target the wrong object in the new view. Camera-only control can let the hands follow the model’s prior instead of the person.

Generated Reality trains the two encoders independently before joint fine-tuning. Simultaneous training is unstable. The paper attributes the difficulty to addition of the two embeddings and ambiguity between motion caused by the hand or camera. This is an instructive result even beyond the particular architecture. Multiple geometric controls do not become compositional merely because they share a tensor shape.

In the qualitative comparison, a hand-only model reaches for a table object instead of the cup on the left after a camera change. Plausible hand shape, and perhaps a pose score, can miss that wrong hand-object relation.

From an offline teacher to a causal student

The high-quality model is still a bidirectional video diffusion model. To make it interactive, the authors distill a conditioned Wan2.2 5B teacher into a causal 5B student with Self Forcing. The AR-Drag section below gives information about this training method. The student generates 12-frame chunks and conditions each frame on the corresponding tracked hand and head data. Recent generated frames supply context for the next chunk.

The deployed loop is split across a Quest 3 and a server with a single H100 GPU. Head and hand states enter a circular buffer. The server reads the latest controls, generates and decodes a chunk, and streams the frames to the headset. The paper gives 11 FPS with 1.4 seconds of latency. The motion conditioning itself adds only 0.002 seconds. Generation and decoding of the 12-frame block are the bottleneck.

These figures distinguish continuous display from quick response. A 1.4-second delay is far from the authors’ sub-20-millisecond target for imperceptible extended-reality (XR) latency. Users can change input continuously and receive generated responses, but responsiveness and stereo rendering still lag conventional headset applications.

Agency evaluation, not only pixel evaluation

I think the user study is the strongest test of the interface claim. Eleven participants push a green button, open a jar and turn a wheel. The study compares joint head/hand control with a baseline that uses head control and text-driven action, without direct hand control.

For this study, the authors give average task accuracy of 71.2% with explicit hand control and 3.0% for the baseline. On a seven-point scale, mean perceived control is 4.21 versus 1.74. They are results for three short tasks, a specified model, and a specified protocol. The evaluation target makes these results useful. The study examines whether a tracked motion lets a person accomplish an intervention, not only whether generated fingers align with held-out joints.

The paper's own limitations stay substantial. More than a quarter of attempts fail. Image quality degrades after a few seconds of rollout. Resolution, latency, stereo, compute efficiency and fidelity lag modern VR. Even a hand approaching the right object can have inconsistent contact between chunks.

I think Generated Reality supplies a systems proof: live joint-level motion can drive causal video and increase perceived agency in these tasks. Reliable physical interaction state for open-ended operation is still unproven.

Case study II: Directing the World composes body and camera

Generated Reality starts from a live first-person user and emphasizes hands. Directing the World starts from a different production problem. Inputs are an initial image, one or more full-body SMPL sequences, and a prescribed camera trajectory. The target is stable world video over 15 to 20 seconds.

Here the motion script and camera path are prescribed. They are not selected after the person sees generated feedback. That is a test of long-horizon composition with multiple actors and scene exploration, rather than a live sensor loop.

A world prior with explicit choreography

The system builds on an existing autoregressive video backbone, Macro-from-Micro Planning (MMPL). The project describes this as a plan-then-populate structure: sparse planning frames, including a terminal anchor, constrain a block before intermediate frames are filled in. A KV cache keeps long-video memory between segments. The architecture is designed to prevent each new block from becoming an unconstrained continuation of the last visible frame.

Human control enters as an SMPL video encoded into a motion latent. A timestep-guided Dynamic Projection adapts the motion condition to the current denoising state. Early in denoising, coarse layout and trajectory are necessary for the model. Subsequent steps can use finer articulation. A fixed projection would ask the same feature transform to serve the two regimes.

The paper also introduces Motion-CFG for motion alignment and supports multiple people with separate motion scripts. This is an important capability of the SMPL interface: a body sequence is entity-like. It can be assigned to an actor, transformed in 3D, and composed with another actor more cleanly than a single foreground optical-flow field.

Train motion first, then add camera control

The method does not train human and camera control as one undifferentiated condition. Stage one learns human motion under static or stable cameras. Stage two adds camera trajectory control on top of the acquired motion prior. Plücker-ray features represent the camera path. A ControlNet-style pathway aligned with block-wise generation supplies these features to the model.

The staged data construction mirrors the architecture. According to the project page, the authors filter 20 million iStock videos into roughly 50,000 curated clips. First come about 20,000 motion-centric clips with strong human motion and limited camera movement. Then come about 30,000 camera-centric clips with larger viewpoint change. Video, text, SMPL, depth or normal cues, and camera estimates are aligned for training.

The split helps to distinguish the causes of pixel motion. Motion-centric clips show body changes under quieter cameras. Camera-centric clips then add viewpoint variation. If the body and camera always moved together, the loss could reward correlation instead of distinct control.

Fast-Slow Memory training adds another separation. New motion and camera-control modules have a higher learning rate. Attention layers that contain the long-video prior have a lower learning rate or stay frozen in the subsequent camera stage. The goal is to adapt the interface and keep the generator's ability to maintain a scene across blocks.

How this interface helps

This is applicable to virtual cinematography. The user prescribes actors and camera. The model generates appearance, lighting, backgrounds and unmodeled dynamics. Separate 3D bodies let choreography survive viewpoint changes and multiple actors share a scene.

Directing the World follows a script that is fixed before generation. It does not react to new movements that a person makes during rollout. Its causal generator and memory still carry earlier frames forward.

The project describes a “fast autoregressive” model with long-horizon exploration, and its 15-to-20-second outputs show long, controlled clips. The project gives no report of the delay from a tracked movement to a displayed frame. It gives no report of a closed human feedback loop. It also gives no report of interactive mid-rollout edits to the SMPL script.

The unresolved consequence problem

SMPL supplies rich body geometry, yet the controlled variable is still primarily human motion. If a scripted hand passes through a prop, the generator must infer the prop response from video priors. If two actors exchange an object, their body sequences do not specify ownership or contact. If a foot lands on a stair, the skeleton constrains the foot trajectory but not the supporting force or whether the stair should stay fixed.

All body-only guidance has these limits. A state for the objects that the body affects is also necessary.

The real-time bridge: AR-Drag and Self Forcing

Bidirectional diffusion can use future frames to change a window. A causal model instead advances from completed history, with a point at each frame/chunk where new controls may enter.

AR-Drag lets users revise sparse trajectories during image-to-video generation and regenerate an unwanted future. It isolates training and serving changes that keep motion guidance editable during rollout. A full human body as control input is not necessary.

AR-Drag first fine-tunes a 1.3B image-to-video model on motion-control data, then distills it into a few-step causal student and uses reinforcement learning. Its trajectory-based reward combines visual quality with motion alignment. This addresses a failure that few-step sampling makes more visible. The model can become fast by smoothing or weakening the requested motion.

AR-Drag’s Self-Rollout trains on the model's own step-by-step samples (ancestral sampling) and model-produced KV history. Subsequent denoising therefore sees context closer to deployment, rather than only clean demonstrations.

Self Forcing attacks the broader version of the same mismatch. The student rolls out on its own generated context during training while it receives distribution-level supervision. It does not condition only on clean ground-truth history. Rolling KV cache makes repeated autoregressive extension efficient. Generated Reality uses this line of work to turn a conditioned bidirectional teacher into its causal student.

My streaming-video article examines these methods in detail. Here, few-step distillation decreases time between controls, causal attention limits inputs to what is available, and self-generated training histories narrow the train/test gap.

A fast model can still ignore the motion. A causal model can still use the wrong state in subsequent frames. Self Forcing can still repeat contact errors. These methods let new controls enter during generation. The control representation and response-based tests determine whether those controls change the right things.

Do tests of the world’s response after the body moves

Pose error is a check of whether the body follows guidance. A large wrist miss clearly indicates weak control, but an accurate wrist says little about the object it should change.

Opening a door. The hand trajectory can align with the handle while the door stays closed, bends, or opens around the wrong hinge. A persistent door angle and a relation between the hand, handle and hinge are necessary for a response-aware model. Once the hand releases, the door should not snap back merely because the pose condition disappeared.

Turning a wheel. A generated hand can rotate around the wheel while the rim texture stays static. The visually plausible foreground hides the missing state transition. The relevant variable is the wheel's angle, coupled to contact over time. For a steering task, an even richer world would connect that angle to downstream motion.

Pulling a rope. The wrist path supplies direction, but rope tension, the segment held by the hand, and the attached object's motion determine the effect. A video prior may produce a generic waving rope that looks dynamic without transmitting the pull.

Petting an animal. There may be no rigid contact state, yet the world should respond: fur deforms, the animal turns, or its posture changes. The exact future is stochastic. The requirement is not one deterministic animation but a plausible response conditioned on the intervention.

These examples suggest three levels of evaluation: visual compliance, causal consequence and persistence.

Visual compliance

At this level, the generated subject, hand, or object trajectory follows the control. Useful measures include joint error, landmark distance, trajectory error, part consistency, identity similarity, and camera-pose error. This is where pose-guided animation has made substantial progress.

For agency, the body must appear at the expected location. Paired videos make that measurable, so alignment is a natural first target.

Causal consequence

Here motion changes object transforms, articulation, deformation, visibility, possession or another agent’s behavior. The affected pixels can be small while the change lasts a long time.

Contact labels can give the times when entities touch and separate, and the parts in contact. Object-level state can record the result. Memory can keep that result after the hand leaves view.

One possible future training tuple is therefore richer than $(\text{video}, \text{pose})$:

where $c$ is human control, $e$ contains interaction events or relations, and $o$ gives information about object-centric state. The generator still learns from pixels, but the auxiliary structure tells it which small changes are consequences rather than incidental texture.

A generative world may stay uncertain and multimodal. The goal is intervention-dependent uncertainty. Turning clockwise should increase the probability of the lid turning, rather than an equally attractive unchanged jar.

Persistence is the hidden test

The stricter test comes after hand and object separate. Does a button stay pressed or a jar stay open? Does an occluded object return unchanged? Does the world remember which actor picked it up?

Persistent world state connects motion guidance to world modeling. Earlier interventions must affect subsequent frames even after their controlling movement leaves view. Memory and object state therefore belong alongside pose conditioning.

How to evaluate the interface

The studies here include offline animation, scripted SMPL, editable sparse trajectories and live XR. Compare the tests in each setup.

Measure the whole loop

For an offline generator, you can measure pose alignment after the generator completes the clip. For a live system, at least four latency timestamps are necessary. These timestamps give the times of sensor observation, control arrival at the generator, the first decoded frame that the control affects, and display. The distribution of those delays is more important than one ideal value. Adaptation to jitter can be more difficult than adaptation to a stable delay.

Give the frequency at which controls can change. One trajectory update for each frame is not the same as a twelve-frame script. Tracking rate and display FPS cannot substitute for that rate.

Separate subject, camera, and world errors

You can use factorial controls for tests of camera disentanglement. While you keep the body fixed, move the camera. While you keep the camera fixed, move the body. Move the body and camera in agreeing and opposing directions. If evaluation only includes natural videos where the two correlate, a model can learn that correlation and get good scores.

Similar interventions are necessary for evaluation of world response. Repeat the same body motion near different objects. While you keep the hand path fixed, change whether contact occurs. Remove the hand after a state change. Then do a persistence test. These counterfactual pairs show whether the model follows geometry or merely generates a familiar action clip.

Use task completion

Generated Reality's button, jar, and wheel tasks are a good pattern because success depends on an outcome. The tasks are narrow enough for evaluators to give a score and embodied enough to show missing control. I recommend that future studies expand the set and keep interpretable completion criteria.

Perceived agency combines latency, predictability, alignment and consequence. Pair it with objective logs. Forgiving animations can feel controllable when they fail precise tasks. Geometrically accurate output can still feel unusable under delay.

Do tests long after the successful frame

Record when identity, geometry, object state and alignment fail during long rollouts. If reversal occurs after two chunks, I recommend that evaluators do not record lasting success for a correct contact.

Select a motion interface

Select the representation for the intended interaction.

For offline character animation, start with an image-aligned pose representation. It gives the generator the strongest spatial bias. It is applicable to a workflow with the full performance available. When tracking or large articulation makes details unstable, add confidence and part-level correspondence.

For full-body choreography across viewpoints, use a 3D parametric body such as SMPL and model camera control separately. This has costs from calibration and encoding complexity but keeps actor motion from collapsing into camera-dependent pixels. Multi-person scenes benefit from entity-level body streams rather than one merged flow field.

For first-person hand interaction, use spatial projection and metric articulation. A 2D skeleton anchors the visible hand. Wrist and finger parameters resolve depth and occlusion. Treat head pose as a first-class camera control. Train it separately. Then train it jointly with the hands.

For open-ended drag or sparse steering, a trajectory interface can be easier to acquire and edit than a full body. It is a good method when the user cares about a region’s destination rather than its exact articulation. Causal few-step generation matters because the trajectory should stay editable during rollout.

For a genuine human-centric interface to a world model, none of those controls is sufficient alone. A causal serving path, an articulated geometric condition, and explicit information about the motion’s effects are necessary for the generator. The most promising stack is:

The first term gives information about the person’s movement. The second receives updates while they act. The third keeps the resulting world changes.

What this changes for research

The research question is how a person can use body movement to steer what a generated scene does next, beyond sharper animation.

Give each component an evaluation based on its contribution to that loop:

• Pose encoders: how much ambiguity they remove
• Faster diffusion: how much it decreases feedback delay
• Memory: whether the model keeps intervention outcomes
• Object representations: whether they protect contact changes that affect too few pixels for generic reconstruction training

This differs from human-to-robot transfer. No new body executes the movement: the controlled target is the generated visual world. The mapping is from tracked human movement to a distribution of future scenes.

Pose estimation, video conditioning, causal generation, interaction understanding and scene memory each supply part of this problem. Together they make a usable control when they agree on coordinates, timing and what the motion changes.

Respond, remember and send feedback

Rich motion sensing can supply more information than prompts or a designed command vocabulary. The output still has to respond.

A 2D skeleton can place a body and leave depth unresolved. SMPL can keep full-body structure and leave unspecified what the body touches. Head and hand parameters can expose first-person intent but leave the generator to invent contact. Causal distillation can decrease latency but also propagate the wrong state faithfully. Each layer resolves one ambiguity but passes another to the next layer.

I would examine whether a human-centric world model responds to an intervention and keeps the outcome. I would also examine whether its displayed response is fast enough to influence the person’s next movement. Under those conditions, motion directs a world. It does not merely supply an animation reference.

@article{ji2026humanmotioncontrolinterfaceforworldmodels,
  title = {Human Motion as a Control Interface for Interactive World Models},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/human-motion-control-interface-for-world-models/}
}

References

1. Xie, Linxi, Lisong C. Sun, Ashley Neall, et al. “Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control.” 2026. Project page.
2. Wang, Haoyuan, Yabo Chen, Haibin Huang, et al. “Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control.” 2026. Project page.
3. Zhao, Kesen, Jiaxin Shi, Beier Zhu, et al. “Real-Time Motion-Controllable Autoregressive Video Diffusion.” 2025. Project page.
4. Wang, Ruiyan, Teng Hu, Kaihui Huang, et al. “PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence.” 2025. Project page.
5. Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. “Adding Conditional Control to Text-to-Image Diffusion Models.” 2023.
6. Hu, Li, Xin Gao, Peng Zhang, et al. “Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation.” 2023.
7. Zhang, Yuang, Jiaxi Gu, Li-Wen Wang, et al. “MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance.” 2024.
8. Wang, Xiang, Shiwei Zhang, Changxin Gao, et al. “UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation.” 2024.
9. Lin, Gaojie, Jianwen Jiang, Jiaqi Yang, et al. “OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models.” 2025.
10. Cheng, Gang, Xin Gao, Li Hu, et al. “Wan-Animate: Unified Character Animation and Replacement with Holistic Replication.” 2025.
11. Zhang, Zhenghao, Junchao Liao, Menghao Li, et al. “Tora: Trajectory-oriented Diffusion Transformer for Video Generation.” 2024.
12. He, Hao, Yinghao Xu, Yuwei Guo, et al. “CameraCtrl: Enabling Camera Control for Text-to-Video Generation.” 2024.
13. Loper, Matthew, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. “SMPL: A Skinned Multi-Person Linear Model.” 2015.
14. Huang, Xun, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion.” 2025.
15. Banerjee, Prithviraj, Sindi Shkodrani, Pierre Moulon, et al. “HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos.” 2024.
16. Fu, Rao, Dingxi Zhang, Alex Jiang, et al. “GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities.” 2024.

Updated 03 November 2026