Human Motion as a Control Interface for Interactive World Models

A person reaches toward a virtual jar, closes two fingers around the lid, and twists. The interesting question is not whether a video model can draw a convincing hand in that pose. It is whether the lid turns, whether the jar stays on the table, and whether the next frame arrives soon enough for the person to adjust their grip.

Text is a low-bandwidth interface

Most video generators are controlled as if the future were a request submitted to a rendering service. A prompt specifies a scene. A first frame fixes an appearance. Perhaps a camera path or a few point trajectories add spatial intent. Then the model produces a clip and the user judges the result after the fact.

That contract works for composition. It is weak for interaction.

"Open the jar" says what should eventually happen but not how the hand approaches, where the fingers close, how the wrist rotates, or when the motion should stop. A keyboard can expose a few directions. A game controller adds continuous axes, but its signals are still an indirect vocabulary that a designer has to map onto the scene. Human motion already contains the spatial structure the experience is trying to express. Head orientation says where attention moves. A wrist pose says where a hand intends to act. A full-body trajectory says how an actor crosses space, balances, turns, and coordinates with other actors.

This makes motion an unusually attractive control source for human-centric world models. It is continuous rather than prompt-level, spatial rather than purely semantic, and available at the same timescale as the experience. It also creates a tempting shortcut: render the tracked skeleton, feed those images into a video diffusion model, and call the result interactive.

The shortcut confuses pose compliance with control.

A model can reproduce the silhouette of a raised arm while ignoring the object the hand was meant to lift. It can follow a walking skeleton while letting the floor slide under the subject. It can draw fingers around a cup without preserving which side of the handle they occupy. It can generate a perfectly plausible camera move that cancels the motion the user intended to see. In each case, the condition is visible in the pixels, yet the person does not control the world in a meaningful sense.

The central claim of this article is therefore stricter:

A pose sequence is not yet a control signal. It becomes one when the generated world responds causally, continuously, and with the right consequences.

Recent work makes this distinction easier to see because the field now spans three different temporal regimes. Systems such as Animate Anyone, MimicMotion, and UniAnimate treat a complete pose sequence as an offline animation specification. Directing the World places structured human motion inside a long-horizon autoregressive world-video model and composes it with camera control. Generated Reality goes further: it streams live head and hand tracking into a causal generator and returns the video to a headset.

These are not successive scores on one benchmark. They expose different meanings of the word control. To understand the progression, it helps to separate the representation, the injection mechanism, the temporal contract, and the consequence the model is expected to preserve.

What makes motion an interface

An interface is a contract between two systems. For a mouse, the contract is not the plastic shell; it is the mapping from physical displacement to a cursor update, with predictable coordinates and delay. Human-motion control needs the same kind of explicit contract.

Let the tracker observe a human state $h_t$ at time $t$. A representation function $\phi$ converts it into a control $c_t$, and a causal generator $G$ advances its internal visual state $s_t$:

The generated frame $x_{t+1}$ returns to the user, who changes the next human state after seeing the response. The last step turns a conditioned generator into a feedback system. A useful interface has to satisfy five requirements along that loop.

The motion must be observable

The system needs a control estimate at the moment the generator consumes it. This sounds obvious, but a motion-capture trajectory cleaned offline is a different input from a headset tracker coping with self-occlusion, lost joints, and sensor jitter. The better the offline reconstruction, the less representative it may be of the live interface.

Observability also depends on the body scale. A camera can recover a coarse 2D torso pose while missing fingertip articulation. A headset knows its own six-degree-of-freedom pose precisely but sees hands only inside a limited field of view. A full-body optical system provides clean geometry but is difficult to deploy casually. The interface begins with whatever uncertainty the sensing arrangement produces.

The coordinate system must be stable

A 2D wrist location means "this pixel." A 3D wrist transform means "this pose in a camera, body, or world frame." An SMPL sequence describes a parameterized body with root motion, joint rotations, and shape. These controls do not remain equivalent when the camera moves.

If a person holds a hand still while turning their head, the hand moves across the image even though its world-space pose may not change. A generator that receives only image-space skeletons can interpret the displacement as hand motion. Conversely, a generator given only world-space joints still needs to know where those joints should project in the current view. Coordinate choice determines which nuisance variables the model must disentangle by itself.

The condition must arrive on time

An offline pose sequence may include the entire future. A live controller cannot. It provides the latest tracked state, perhaps with a small prediction to compensate for latency. A model trained with future-visible pose tokens can appear accurately controlled in a benchmark and fail when those tokens are unavailable online.

The relevant delay is not only network inference time. It is the sum of sensing, tracking, buffering, encoding, denoising, decoding, transport, and display. A generator at high throughput can still feel unresponsive if it waits for a large chunk before displaying the first frame.

Controls must compose without erasing one another

Human motion rarely acts alone. Text specifies identity and setting. A reference image fixes appearance. Camera motion defines viewpoint. Previous frames carry scene state. Adding a new condition changes how the model distributes attention among all of them.

The conflict is especially sharp between subject and camera motion. In pixels, both appear as optical flow. If a dancer moves left while the camera pans right, the generator must allocate part of the observed displacement to the body and part to the world. Conditioning modules that work separately can interfere when activated together.

The motion must produce a consequence

This is the requirement that separates character animation from world control. When a tracked hand pushes a button, the button should depress and remain in the new state. When it turns a steering wheel, the wheel should rotate around the right axis rather than merely being repainted under the palm. When a person walks behind a table, the occlusion order should change and then remain consistent.

The output is not required to obey a hidden simulator perfectly. It is required to maintain a coherent intervention-to-outcome relation. Without that relation, motion controls a foreground performance while the world remains a generative backdrop.

These five requirements give us a more useful comparison than visual quality alone. For each method, I will ask: what signal enters, which invariant it preserves, where it is injected, when it is available, what portion of the world it controls, and which error it sends downstream.

The representation ladder

Human motion can enter a generator at several levels of abstraction. Moving upward in geometric structure does not produce a universally better interface. Each rung removes some ambiguity and introduces a different dependency.

2D skeletons: strong pixels, weak depth

The simplest pose condition is a rendered skeleton or heatmap. Each joint becomes a point or Gaussian channel; bones connect selected pairs. Because the representation lives in image coordinates, a convolutional or video encoder can align it directly with the target pixels. It says, with little translation overhead, "the elbow should appear here."

This spatial alignment is one reason pose-guided animation became practical before general 3D-aware control. ControlNet established a reusable pattern for attaching spatial conditions such as edges, depth, segmentation, and human pose to a powerful pretrained diffusion backbone. Animate Anyone combined a pose guider with a dedicated ReferenceNet and temporal layers, separating the problems of following motion, preserving appearance, and maintaining inter-frame continuity.

The weakness is equally direct. A 2D skeleton cannot distinguish a hand moving toward the camera from a hand growing in the image. Crossing fingers can collapse multiple 3D configurations to the same projection. Joints outside the frame disappear. And when the camera moves, the projected skeleton changes even if the body remains fixed in world coordinates.

Confidence becomes part of the interface. MimicMotion uses confidence-aware pose guidance and amplifies losses in relevant regions, acknowledging that a detected joint should not be treated as equally reliable in every frame. Its progressive latent fusion supports longer outputs, but the full driving pose sequence is still supplied as an offline specification. It improves the animation contract; it does not make the sensing and generation loop causal.

DensePose-like controls add surface correspondence between body regions and image locations. They give the generator more information than sparse bones, especially around torso shape and limb extent. They also bind the control more tightly to a particular visible surface and remain vulnerable to occlusion and camera movement. For loose clothing, props, hair, and hands, a dense body map still leaves important motion outside the template.

3D joints and SMPL: a body that survives the camera

Three-dimensional joints replace image locations with a kinematic configuration in a chosen coordinate frame. A parametric model such as SMPL goes further by representing pose-dependent body geometry and identity-dependent shape through a compact, animatable body model.

For full-body control, this is a useful interface because it separates the actor from the camera. Root translation can describe where the person moves through the scene. Joint rotations describe articulation. The same motion can be reprojected from a new viewpoint or assigned to more than one generated subject. The control is no longer synonymous with one set of target pixels.

That abstraction has a cost. A generator ultimately produces images, so 3D parameters must be encoded, rendered, projected, or converted into tokens that can influence spatial features. Calibration errors between the SMPL coordinate frame and the generated camera become visible as floating feet or sliding bodies. SMPL also models the human body more naturally than loose garments, held objects, or the exact surface of a contacting hand. It gives a stable body scaffold, not a complete description of interaction.

The difference matters when a full-body motion contains an outcome that is not on the body. A tennis swing represented by SMPL can locate shoulders, elbows, and wrists but not the racket face or ball contact unless those entities receive their own state. A sitting motion can place the pelvis correctly while the generated chair deforms or shifts. Moving to 3D removes viewpoint ambiguity; it does not automatically add object causality.

Head and hand parameters: interaction at first-person scale

In a headset, the most informative signals are often not a full reconstructed body. The camera pose is already measured by the device. The hands can be represented by a wrist transform plus joint articulation. This is a compact, high-rate description of what the user can see and act with.

The head pose carries two meanings at once: it moves the camera, and it reveals attention. Hand parameters carry articulation in metric depth, including configurations that overlap in the image. For first-person interaction, this is a more direct control vocabulary than a distant full-body skeleton.

The representation is also unforgiving. Hand-object interaction depends on small spatial differences. Tracking noise that would be harmless for a dancing silhouette can change whether a fingertip appears inside a lid or beside it. Hands frequently leave the headset's view, occlude each other, or approach the image boundary. The generator has to translate joint rotations into a visually plausible hand while preserving the intended target object.

Datasets illustrate the sensing burden. HOT3D provides egocentric multi-view streams with tracked 3D hands, objects, and cameras, including UmeTrack and MANO-format annotations. GigaHands expands the diversity of bimanual activity with 34 hours, 56 subjects, 417 objects, and 14,000 motion clips, according to its authors. These datasets do not solve interactive generation, but they make the hand-object geometry that such an interface needs more learnable.

Hybrid 2D-3D control: two imperfect views of the same hand

Hybrid control combines a rendered, image-aligned skeleton with the underlying 3D parameters. The 2D branch tells the generator where the hand should appear. The 3D branch disambiguates depth, wrist rotation, and finger articulation. Neither is asked to carry the entire interface.

This is not merely feature accumulation. The two signals have complementary error modes. A correct 3D pose projected with a slightly wrong camera may be spatially misaligned. A 2D skeleton can be perfectly aligned while hiding whether one finger crosses in front of another. Fusion gives the model a chance to use image-space evidence for placement and parametric evidence for structure.

Generated Reality's ablation makes this concrete. The authors compare hand-pose parameter injection, binary masks, skeleton video, and a hybrid of skeleton video plus 3D parameters on HOT3D. Within their setup, the hybrid achieves the strongest reported hand-accuracy metrics while retaining competitive video quality. This is evidence for the complementarity of the signals in that model and dataset, not a universal ranking of pose representations.

Learned motion features: useful compression, uncertain semantics

A learned encoder can turn poses, optical flow, or tracked parameters into continuous motion features. Such features can smooth sensor noise, compress long sequences, and adapt to the backbone's latent space. They are attractive when the raw geometry and video tokens have very different dimensionality.

But a learned feature weakens the external contract. If one latent direction mixes wrist rotation with camera motion, a downstream user cannot easily correct it. If the feature is learned only to reconstruct video, it may preserve visually predictive motion while discarding contact states that rarely change many pixels. Learned representations work best when the training objective names the behavior the interface must retain: temporal alignment, part correspondence, object response, or another measurable consequence.

The representation ladder therefore ends in a tradeoff, not a winner. Image-space controls are easy to align and hard to disambiguate. Parametric controls are geometrically stable and harder to inject. Hybrid controls spend more compute and data to reduce both errors. Learned features can absorb the mess, but only to the extent that the supervision defines what should survive.

How motion enters a video model

A good representation can still fail if it enters the generator in the wrong place. Conditioning is a routing problem: the motion signal has to influence spatial layout, temporal evolution, and the denoising trajectory without erasing appearance or camera intent.

Concatenation and residual control

The most literal strategy concatenates a pose image or latent with the noisy video latent. The first projection layer learns how to mix them. The alignment is strong because matching spatial positions meet early. The cost is architectural: changing the number of channels modifies the input layer, and a strong condition can dominate features that the pretrained model already uses well.

ControlNet-style branches reduce that disruption. A copied or lightweight control pathway processes the spatial condition and adds residuals into selected backbone blocks. Zero-initialized projections allow the new path to begin without immediately changing the pretrained network. For video, the branch must also respect temporal alignment; a per-frame pose map is not enough if its features flicker or arrive with a different frame schedule.

Token addition and concatenation

Diffusion transformers make conditions look like token operations. Concatenation appends motion tokens to the video sequence and lets attention decide where to use them. This is flexible, but it increases sequence length and does not guarantee that a hand parameter at time $t$ aligns with the correct spatial region.

Token addition projects the control into the same shape as video patch tokens and adds the vectors element by element. It is cheap and preserves the token count. It also demands a meaningful alignment between control embeddings and video tokens. In Generated Reality, the authors evaluate token concatenation, token addition, adaptive layer normalization, and cross-attention for hand-pose parameters; token addition is the strongest of those parameter-only variants on their hand-accuracy metrics. Their final hybrid uses channel-wise skeleton-video conditioning plus added hand-parameter features.

Cross-attention and adaptive normalization

Cross-attention keeps control tokens separate. Video features query motion features as needed, which is useful when the control is not naturally image-shaped or when the number of entities varies. The module can be placed after selected transformer blocks, limiting how much of the backbone changes.

The freedom can become a data problem. Attention has to discover the association between a joint parameter and the correct spatial region. On small tracking datasets, that mapping may be harder to learn than a spatially rendered skeleton. AdaLN has a similar global character: scale and shift parameters modulate block activations efficiently, but the modulation does not by itself state where the fingertip belongs.

The practical question is not which operator is most expressive. It is where alignment is already known. If a condition is a pixel-aligned skeleton, early spatial fusion is a useful bias. If it is a set of 3D entities or motion tokens, cross-attention may be more natural. If it is a compact global mode such as camera style or timestep, adaptive normalization can be sufficient. A system that ignores this distinction forces data to relearn geometry the representation already contained.

Part-aware temporal coherence

PoseAnything exposes another conditioning problem: following a skeleton is not enough to preserve the subject attached to it. Large articulation changes move corresponding body parts far apart in pixel space. Standard temporal attention can confuse similar regions or let fine details dissolve.

The method accepts a reference image and an arbitrary skeletal sequence, including non-human skeletons. It encodes reference and pose inputs into latent features and concatenates them by channel. Its Part-aware Temporal Coherence Module segments the skeleton into parts, dilates those segments into masks, matches corresponding parts across frames using attention patterns, and performs cross-attention between the matched regions. The interface is not only "place joints here"; it adds "this moving region is the same semantic part as that earlier region."

PoseAnything also highlights control interference. It places subject-motion guidance and camera-motion guidance in different classifier-free-guidance anchors so one signal does not simply absorb the other. Whatever one thinks of the exact mechanism, the diagnosis is important: body and camera motion are not independent in image space. A controllable generator needs an explicit policy for composing them.

Why controls fight

Suppose the subject moves ten pixels right between frames. The same observed flow could come from the subject, a leftward camera pan, a deformation of the body, or a combination. Text may also imply motion: "a runner sprinting" pushes the model toward one distribution even if the supplied skeleton is nearly static. A reference image may resist the large viewpoint change requested by the camera path because the unseen side of the person has no visual evidence.

Conditioning modules therefore allocate authority. A high guidance scale can improve pose alignment while damaging identity or texture. Strong camera control can keep the desired trajectory and make the subject slide. Strong reference attention can preserve clothing and reduce articulation. Compositional control is not achieved by checking that every condition has a corresponding encoder. It requires training examples and objectives in which the conditions vary independently enough for the model to learn their roles.

This is why staged training appears repeatedly. Learn a stable motion control under static cameras, then add camera movement. Train camera and hand encoders independently, then tune them jointly. Freeze the expensive prior while adapting new control paths. These choices are responses to one fact: the video prior already explains motion, and every explicit controller must negotiate with what the model would have generated on its own.

Three temporal contracts

The word real-time often enters a paper after the model has become faster. Interaction demands a more precise question: when is the control known, when can it change, and when does its effect become visible?

Contract one: the future is already known

Offline pose-guided animation receives a full reference sequence $c_{1:T}$. The model can process all frames jointly or divide them into overlapping windows. Bidirectional temporal attention can use a later pose to resolve an earlier ambiguity. Progressive latent fusion can smooth the boundaries between windows. The output may be a minute long, but no one can change frame 120 in response to frame 119 because the motion script is fixed before generation begins.

This is a valuable contract for film, avatar animation, fashion, dance, and performance transfer. UniAnimate, for example, unifies reference, pose, and noise features and uses first-frame conditioning to extend generation iteratively. Wan-Animate combines spatially aligned skeleton signals and facial features for character animation and replacement. OmniHuman-1 studies scaling one-stage human animation with mixed motion-related conditions. None needs to be interactive to be useful.

The danger is terminological. A generator that renders frames quickly after receiving the full sequence may have high throughput. It is not a live control interface if changing the next movement requires restarting or revising an already planned clip.

Contract two: the future arrives in blocks

A causal or chunk-wise generator produces the next block from previous frames and a control segment. It can reuse cached state and avoid denoising an entire long clip. New controls may enter at chunk boundaries. Long-horizon autoregression becomes possible because the model advances rather than recomputes the full past.

This contract has a control horizon. If the model generates twelve frames at once, it needs controls for those twelve frames before the chunk begins. A late hand movement cannot affect the frames already being decoded. Smaller chunks reduce that delay but increase overhead and may weaken temporal coherence. Larger chunks amortize computation but make the interface feel more buffered.

Chunk-wise systems can be excellent directors. A preauthored SMPL motion and camera path can guide each new block while a memory mechanism maintains the scene. They become interactive only if the control producer and the generator exchange updates during the rollout.

Contract three: the person closes the loop

Live control adds sensing and feedback. At time $t$, the system reads a recent human state, produces a response, displays it, and lets the person choose $h_{t+1}$ after seeing what happened. The control is endogenous to the generated video because the video changes the user's next action.

That loop introduces four rates:

• tracking rate: how frequently the body, head, or hands are estimated;
• control consumption rate: how frequently the generator accepts a new condition;
• display rate: how frequently completed frames reach the user;
• response latency: how long an observed movement takes to affect a visible frame.

Frames per second describes only one part. Eleven FPS can coexist with 1.4 seconds of response latency if the system waits for a chunk. A model may generate faster than display speed but accept control only every half-second. A tracker may run at headset rate while the generator consumes a downsampled trajectory. For interaction, these differences shape agency more than a single throughput number.

The user also becomes an adaptive controller. People slow their hands when latency is visible, exaggerate a motion when tracking is uncertain, and stop when the world responds unexpectedly. A closed-loop evaluation therefore cannot be replaced entirely by pose-alignment metrics on a fixed dataset. The person's behavior is part of the system.

With these temporal contracts in place, we can read the two most revealing recent systems without collapsing them into one category. Generated Reality is a live first-person loop. Directing the World is a compositional long-horizon director. Both use human motion, but they ask it to do different jobs.

Case study I: Generated Reality closes the loop

Generated Reality begins from a constraint familiar to anyone who has tried a generative game demo: text and keyboard controls are too coarse for hands. "Turn the wheel" cannot specify which hand reaches first, how the wrists rotate, or how the grip changes. A conventional virtual-reality application solves this with a 3D scene graph, collision geometry, a renderer, and interaction code written for the wheel. The paper asks whether a video world model can synthesize the experience without those manually constructed assets while still responding to tracked motion.

The system uses a Meta Quest 3 to track the user's head and hands. The head provides a six-degree-of-freedom camera pose. Each hand is represented with UmeTrack parameters: a wrist transformation plus 20 joint angles. The generator receives both a 2D projection of the hand skeleton and those 3D parameters.

This is the hybrid interface from the representation ladder in operational form. The skeleton frames enter through a VAE-aligned visual branch. A lightweight motion encoder maps the hand-pose parameters into token features. A camera encoder converts head poses into per-frame Plücker embeddings. The resulting features are combined by token addition before the diffusion-transformer blocks.

Why head and hand control have to be learned together

The headset camera moves with the user's head. That motion changes every projected hand position as well as the background. A hand-only controller can place the hand correctly relative to the old view while reaching toward the wrong object after a head turn. A camera-only controller can move the view correctly while allowing the generated hands to follow the model's prior rather than the user.

Generated Reality trains the two encoders independently before joint fine-tuning because simultaneous training is unstable. The paper attributes the difficulty to both embeddings entering through addition and to ambiguity between motion caused by the hand and motion caused by the camera. This is an instructive result even beyond the particular architecture. Multiple geometric controls do not become compositional merely because they share a tensor shape.

The authors' qualitative comparison gives a useful failure example. A hand-only model reaches toward an object on the table instead of the cup on the left after the camera changes. The hand can remain plausible, and a hand-pose metric may not fully express the error. The failure is relational: the generated hand no longer acts on the object the user sees as the target.

From an offline teacher to a causal student

The high-quality model is still a bidirectional video diffusion model. To make it interactive, the authors distill a conditioned Wan2.2 5B teacher into a causal 5B student following Self-Forcing. The student produces 12-frame chunks and conditions each frame on the corresponding tracked hand and head data. Recent generated frames serve as context for the next chunk.

The deployed loop is split across a Quest 3 and a server with a single H100 GPU. Head and hand states enter a circular buffer. The server reads the latest controls, generates and decodes a chunk, and streams the frames back to the headset. The paper reports 11 FPS with 1.4 seconds of latency; adding the motion conditioning itself accounts for only 0.002 seconds. The bottleneck is generating and decoding the 12-frame block.

Those numbers illustrate why throughput and control latency must remain separate. Eleven new frames per second is enough to display continuous output. A 1.4-second sensor-to-response delay is still far from the sub-20-millisecond target the authors name for imperceptible XR latency. The system is interactive in the meaningful sense that users can continuously change the input and receive a generated response. It is not yet comparable to the responsiveness or stereo rendering of a conventional headset application.

Evaluating agency instead of only pixels

The user study is the strongest part of the interface claim because it measures what people can do inside the loop. Eleven participants perform three tasks: push a green button, open a jar, and turn a steering wheel. Each task is attempted with joint head-and-hand control and with a baseline that keeps head control but removes direct hand control, relying on text to produce the desired action.

Within this study, the authors report average task accuracy of 71.2% with explicit hand control and 3.0% for the baseline. On a seven-point scale, mean perceived control is 4.21 versus 1.74. These are not general success rates for generated environments. They are results for three short tasks, a particular model, and a particular protocol. What makes them useful is the evaluation target: the study asks whether a tracked motion lets a person accomplish an intervention, not only whether generated fingers align with held-out joints.

The remaining gap is visible in the absolute numbers and in the paper's limitations. Users still fail more than a quarter of the attempts. Image quality degrades after a few seconds of autoregressive rollout. Resolution, latency, stereo, compute efficiency, and visual fidelity lag modern VR systems. A generated hand can approach the correct object while contact changes unpredictably from one chunk to the next.

Generated Reality is therefore best understood as a systems proof. It shows that joint-level live motion can cross the boundary into a causal video generator, and that people experience more agency when it does. It does not yet show that a video model can maintain a reliable physical interaction state for open-ended use.

Case study II: Directing the World composes body and camera

Generated Reality starts from a live first-person user and emphasizes hands. Directing the World starts from a different production problem: given an initial image, one or more full-body SMPL sequences, and a prescribed camera trajectory, generate a stable world video over 15 to 20 seconds.

The distinction matters. The motion is a script rather than a live sensor stream. The camera path is known rather than chosen after observing the previous generated block. That makes the method less interactive, but it permits a richer test of long-horizon composition: multiple actors can follow articulated motion while the camera explores the scene.

A world prior with explicit choreography

The system builds on an MMPL autoregressive backbone. The project describes this as a plan-then-populate structure: sparse planning frames, including a terminal anchor, constrain a block before intermediate frames are filled in. A KV cache carries long-video memory across segments. The architecture is designed to prevent each new block from becoming an unconstrained continuation of the last visible frame.

Human control enters as an SMPL video encoded into a motion latent. A timestep-guided Dynamic Projection adapts the motion condition to the current denoising state. Early in denoising, the model needs coarse layout and trajectory. Later steps can use finer articulation. A fixed projection would ask the same feature transform to serve both regimes.

The paper also introduces Motion-CFG for motion alignment and supports multiple people with separate motion scripts. This is an important capability of the SMPL interface: a body sequence is entity-like. It can be assigned to an actor, transformed in 3D, and composed with another actor more cleanly than a single foreground optical-flow field.

Decouple, then integrate

The method does not train human and camera control as one undifferentiated condition. Stage one learns human motion under static or stable cameras. Stage two adds camera trajectory control on top of the acquired motion prior. The camera path is represented through Plücker-ray features and injected by a ControlNet-style pathway aligned with block-wise generation.

The staged data construction mirrors the architecture. According to the project page, the authors filter 20 million iStock videos into roughly 50,000 curated clips: about 20,000 motion-centric clips with strong human motion and limited camera movement, followed by about 30,000 camera-centric clips with larger viewpoint change. Video, text, SMPL, depth or normal cues, and camera estimates are aligned for training.

This split is more than dataset housekeeping. If every energetic body motion co-occurs with handheld camera movement, the model can minimize training loss without learning which condition owns which pixels. Motion-centric data creates examples where the body changes and the camera stays relatively quiet. Camera-centric data then teaches viewpoint variation after the motion mapping is established.

Fast-Slow Memory training adds another separation. New motion and camera-control modules learn with a higher rate, while attention layers that contain the long-video prior update more slowly or remain frozen in the later camera stage. The goal is to adapt the interface without destroying the generator's ability to maintain a scene across blocks.

What this interface buys

The result is closer to a virtual cinematography tool than a headset loop. A director can prescribe where actors move and where the camera goes while the video model supplies appearance, lighting, background detail, and unmodeled dynamics. Because the body control is three-dimensional and separate from the camera, the same choreography can survive large perspective changes. Multi-person control can organize several actors inside one persistent scene.

The method also reveals a useful role for world models that is not identical to user interaction. A world model can be directable over a long horizon without being reactive to an unknown next input. This still demands causal generation and memory because later blocks depend on earlier generated state. But the controller can plan the whole sequence in advance.

The boundary should remain explicit. The project describes a "fast autoregressive" model and presents long-horizon world exploration. That does not establish a live tracking-to-display latency or a closed human feedback loop. Its 15-to-20-second outputs are evidence of controlled duration, not a claim that a person can change an SMPL trajectory mid-rollout with imperceptible delay.

The unresolved consequence problem

SMPL provides rich body geometry, yet the controlled variable remains primarily human motion. If a scripted hand passes through a prop, the generator must infer the prop response from video priors. If two actors exchange an object, their body sequences do not specify ownership or contact. If a foot lands on a stair, the skeleton constrains the foot trajectory but not the supporting force or whether the stair should remain fixed.

This is not a criticism unique to Directing the World. It is the boundary of body-centric guidance. The method shows how to preserve choreography and camera exploration over a longer generated world. The next interface needs explicit state for the parts of the world changed by the choreography.

The real-time bridge: AR-Drag and Self-Forcing

The step from a controllable clip to an interactive generator is partly architectural. A bidirectional diffusion model denoises a temporal window jointly. It can exploit future frames and revise the whole clip, but it has no natural moment when the current output is final and a new control can enter. A causal model generates from completed history and can expose a control point at each frame or chunk.

AR-Drag studies this transition for sparse motion controls rather than a full human body. Its user can update trajectories during image-to-video generation and regenerate an unsatisfactory future without recomputing a complete bidirectional clip. The method is useful here as a bridge: it isolates the serving and training machinery required for a motion condition to remain editable during rollout.

AR-Drag first fine-tunes a 1.3B image-to-video model on motion-control data, then distills it into a few-step causal student and applies reinforcement learning. Its trajectory-based reward combines visual quality with motion alignment, addressing a failure that few-step sampling makes more visible: the model can become fast by smoothing or weakening the requested motion.

The training history is equally important. An autoregressive model deployed on its own frames should be trained under the distribution created by its own frames. AR-Drag's Self-Rollout preserves a step-by-step ancestral process and carries model-generated history in KV memory. That makes the state seen by later denoising decisions closer to the state present at inference.

Self Forcing attacks the broader version of the same mismatch. Instead of conditioning only on clean ground-truth history, the student rolls out on its own generated context during training while receiving distribution-level supervision. Rolling KV cache makes repeated autoregressive extension efficient. Generated Reality uses this line of work to turn a conditioned bidirectional teacher into its causal student.

These mechanisms deserve only a short treatment here because they are covered in detail in my article on streaming autoregressive video generation. For the motion-interface argument, their role is precise: few-step distillation reduces the time between control updates; causal attention defines which inputs are legally available; self-generated training history reduces the gap between a clean demonstration and a live rollout.

None guarantees agency by itself. A fast model can ignore the motion. A causal model can preserve the wrong state. A self-forced model can faithfully reproduce its own contact error. Real-time generation is the transport layer of the interface. The semantics still come from the control representation and the consequences used to train or evaluate it.

Pose following is not world response

The easiest metric for motion control compares a generated pose with the conditioning pose. It is useful. If the wrist misses by a large margin, the system is not controlled. But pose accuracy measures the commanded body, not the state changed by that body.

Consider four apparently simple controls.

Opening a door. The hand trajectory can align with the handle while the door stays closed, bends, or opens around the wrong hinge. A response-aware model needs a persistent door angle and a relation between the hand, handle, and hinge. Once the hand releases, the door should not snap back merely because the pose condition disappeared.

Turning a wheel. A generated hand can rotate around the wheel while the rim texture remains static. The visually plausible foreground hides the missing state transition. The relevant variable is the wheel's angle, coupled to contact over time. For a steering task, an even richer world would connect that angle to downstream motion.

Pulling a rope. The wrist path supplies direction, but rope tension, the segment held by the hand, and the attached object's motion determine the effect. A video prior may produce a generic waving rope that looks dynamic without transmitting the pull.

Petting an animal. There may be no rigid contact state, yet the world should respond: fur deforms, the animal turns, or its posture changes. The exact future is stochastic. The requirement is not one deterministic animation but a plausible response conditioned on the intervention.

These examples suggest two levels of success.

Visual compliance

At this level, the generated subject, hand, or object trajectory follows the control. Useful measures include joint error, landmark distance, trajectory error, part consistency, identity similarity, and camera-pose error. This is where pose-guided animation has made substantial progress.

Visual compliance is necessary for agency because a user's body must appear where they expect. It is also measurable from paired videos. That makes it the natural first target for a new conditioning method.

Causal consequence

At this level, the controlled motion changes another state in a way that remains coherent. The change can involve an object transform, articulation, deformation, visibility, ownership, or the behavior of another agent. The important variables may be small in pixel area and long in duration.

A contact event is a useful intermediate representation. Instead of asking the model to infer every interaction from raw pose, supervision can identify when two entities touch, which parts participate, and when the contact breaks. Object-centric state can record the outcome. Persistent memory can carry the change after the controlling hand leaves the frame.

One possible future training tuple is therefore richer than $(\text{video}, \text{pose})$:

where $c$ is human control, $e$ contains interaction events or relations, and $o$ describes object-centric state. The generator still learns from pixels, but the auxiliary structure tells it which small changes are consequences rather than incidental texture.

This does not require turning the video model into a conventional physics engine. A generative world may remain uncertain and multimodal. The goal is to make the uncertainty conditional on the intervention. When the hand turns clockwise, futures in which the lid turns with it should receive more probability than equally pretty futures in which nothing happens.

Persistence is the hidden test

Many interaction demos look convincing for a second because the hand and object overlap. The stricter test begins after separation. Does the button stay pressed? Does the jar remain open? Does an occluded object return with the same state? Does the world remember which actor picked it up?

Persistence connects motion control to world modeling. An animation model can satisfy each frame's pose independently. A world model must carry the consequences of earlier interventions into frames that no longer contain the original control evidence. This is why memory design and object state belong in the same conversation as pose conditioning.

How to evaluate the interface

No single benchmark currently spans offline character animation, long-horizon SMPL control, sparse real-time trajectories, and live XR interaction. A responsible comparison should therefore organize evidence by contract rather than sort reported numbers.

The table is deliberately not a leaderboard. It says where each interface sits and what its evaluation can support.

Measure the whole loop

For an offline generator, pose alignment can be measured after the clip is complete. A live system needs at least four latency timestamps: sensor observation, control arrival at the generator, first affected decoded frame, and display. Reporting the distribution of those delays matters more than one ideal value. Jitter can be harder to adapt to than a stable delay.

Control bandwidth should also be explicit. A system accepting a new trajectory every frame exposes a different interface from one accepting a twelve-frame script. Tracking rate and display FPS should not stand in for control-update rate.

Separate subject, camera, and world errors

Camera disentanglement can be tested with factorial controls: hold the body fixed and move the camera; hold the camera fixed and move the body; move both in agreeing and opposing directions. If evaluation only includes natural videos where the two correlate, a model can score well by learning that correlation.

World response needs similar interventions. Repeat the same body motion near different objects. Hold the hand path fixed and vary whether contact occurs. Remove the hand after a state change and test persistence. These counterfactual pairs reveal whether the model follows geometry or merely generates a familiar action clip.

Use task completion without pretending it is universal

Generated Reality's button, jar, and wheel tasks are a good pattern because success depends on an outcome. The tasks are narrow enough to score and embodied enough to expose missing control. Future studies should expand the set while preserving interpretable completion criteria.

Perceived agency is also meaningful. Users integrate latency, predictability, visual alignment, and consequence into one judgment. It should accompany, not replace, objective logs. A system can feel controllable because it produces forgiving animations while failing a precise task; it can also achieve geometric accuracy and feel unusable because of delay.

Test long after the successful frame

Long-rollout evaluation should record when identity, geometry, object state, and control alignment fail, not only average video quality over the full sequence. The moment after contact is especially diagnostic. If the object changes correctly and then reverts two chunks later, a frame-level success metric will overstate the interface.

Choosing a motion interface

The right representation follows from the interaction contract.

For offline character animation, start with an image-aligned pose representation. It gives the generator the strongest spatial bias and fits a workflow in which the full performance is available. Add confidence and part-level correspondence when tracking or large articulation makes details unstable.

For full-body choreography across viewpoints, use a 3D parametric body such as SMPL and model camera control separately. This costs calibration and encoding complexity but keeps actor motion from collapsing into camera-dependent pixels. Multi-person scenes benefit from entity-level body streams rather than one merged flow field.

For first-person hand interaction, use both spatial projection and metric articulation. A 2D skeleton anchors the visible hand. Wrist and finger parameters resolve depth and occlusion. Treat head pose as a first-class camera control and train it both separately and jointly with the hands.

For open-ended drag or sparse steering, a trajectory interface can be easier to acquire and edit than a full body. It works well when the user cares where a region goes but not about exact articulation. Causal few-step generation matters because the trajectory should remain editable during rollout.

For a genuine human-centric world-model interface, none of those controls is sufficient alone. The generator needs a causal serving path, an articulated geometric condition, and an explicit account of what the motion changes. The most promising stack is:

The first term tells the model what the person did. The second makes the signal available while the person is still acting. The third prevents the world from treating the action as foreground choreography with no memory.

The application narrative

This direction sits at a useful intersection of dynamic visual modeling and interactive intelligence. The technical question is not only how to generate sharper human motion. It is how to construct a visual model whose future can be addressed through the body.

That reframing changes what counts as progress. Better pose encoders matter when they reduce ambiguity at the control boundary. Faster diffusion matters when it shortens the feedback loop. Long-video memory matters when it preserves the result of an earlier intervention. Object-centric representations matter when the pixels changed by contact are too small or too rare for a generic reconstruction loss to protect.

It also explains why this topic should not be folded into human-to-robot transfer. No new body is being asked to execute the human motion. The target is the generated visual world itself. The relevant mapping is from a person's tracked movement to a controllable distribution over future scenes.

The research opportunity is to join lines of work that are usually evaluated separately: pose estimation, controllable video diffusion, causal generation, interaction understanding, and persistent scene modeling. Each already supplies a piece. The interface appears only when the pieces agree on coordinates, timing, and consequence.

Conclusion

Human motion is richer than a prompt and more natural than a hand-authored command vocabulary. But richness at the sensor does not guarantee control at the output.

A 2D skeleton can place a body without resolving depth. SMPL can preserve full-body structure without describing what the body touches. Head and hand parameters can expose first-person intent while leaving the generator to invent contact. Causal distillation can reduce latency while faithfully propagating the wrong state. Every layer solves one ambiguity and leaves another for the next.

The decisive step is to stop treating pose matching as the endpoint. A human-centric world model should respond to an intervention, retain its outcome, and return that response quickly enough to influence what the person does next. When those conditions hold, motion is no longer an animation reference. It is the language through which the world is controlled.

@article{ji2026humanmotioncontrolinterfaceforworldmodels,
  title = {Human Motion as a Control Interface for Interactive World Models},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {July},
  url = {https://fusheng-ji.github.io/blog/posts/human-motion-control-interface-for-world-models/}
}

References

1. Xie, Linxi, Lisong C. Sun, Ashley Neall, et al. “Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control.” 2026. Project page.
2. Wang, Haoyuan, Yabo Chen, Haibin Huang, et al. “Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control.” 2026. Project page.
3. Zhao, Kesen, Jiaxin Shi, Beier Zhu, et al. “Real-Time Motion-Controllable Autoregressive Video Diffusion.” 2025. Project page.
4. Wang, Ruiyan, Teng Hu, Kaihui Huang, et al. “PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence.” 2025. Project page.
5. Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. “Adding Conditional Control to Text-to-Image Diffusion Models.” 2023.
6. Hu, Li, Xin Gao, Peng Zhang, et al. “Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation.” 2023.
7. Zhang, Yuang, Jiaxi Gu, Li-Wen Wang, et al. “MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance.” 2024.
8. Wang, Xiang, Shiwei Zhang, Changxin Gao, et al. “UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation.” 2024.
9. Lin, Gaojie, Jianwen Jiang, Jiaqi Yang, et al. “OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models.” 2025.
10. Cheng, Gang, Xin Gao, Li Hu, et al. “Wan-Animate: Unified Character Animation and Replacement with Holistic Replication.” 2025.
11. Zhang, Zhenghao, Junchao Liao, Menghao Li, et al. “Tora: Trajectory-oriented Diffusion Transformer for Video Generation.” 2024.
12. He, Hao, Yinghao Xu, Yuwei Guo, et al. “CameraCtrl: Enabling Camera Control for Text-to-Video Generation.” 2024.
13. Loper, Matthew, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. “SMPL: A Skinned Multi-Person Linear Model.” 2015.
14. Huang, Xun, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion.” 2025.
15. Banerjee, Prithviraj, Sindi Shkodrani, Pierre Moulon, et al. “HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos.” 2024.
16. Fu, Rao, Dingxi Zhang, Alex Jiang, et al. “GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities.” 2024.