Perception requirements for action

An agent turns away from a drawer, returns and tries to pull it open. It must find the handle and keep information that shows if the drawer moved. It must predict the result of a pull and select an action that its body can execute. A coherent image gives only part of the solution.

I think video world models and 3D geometry solve different parts of this problem. Video supplies temporal experience, and 3D keeps geometric consistency between views. The proposed ImageNet moment depends on one test: does pretraining make control better on held-out tasks and robot bodies?

Vincent Sitzmann’s The flavor of the bitter lesson for computer vision is important to me. The essay’s criterion for perception is how much it helps an agent execute actions. Intermediate tasks include reconstruction, segmentation, tracking and pose estimation. These tasks divided the larger goal into components for system construction.

I use four requirements to judge perception by how much it helps action:

These requirements interact. Explicit 3D can represent spatial state, but it does not supply action consequences. For each method, examine the requirement it helps the model learn. Examine the evidence for that claim. Find the parts that stay untested.

Recover geometry that agrees across views

My earlier work involved 3D reconstruction, scene decomposition and long-term video tracking. Explicit structure was the most workable way I found to keep scenes and humans consistent, interpretable and usable downstream. Different images could describe the same objects instead of making unrelated predictions.

For a stationary scene point $X$, the basic geometric relation is:

Here $u_i$ gives the location of the point in view $i$. The transform $T_i$ gives camera coordinates, $K_i$ holds camera intrinsics, and $\pi$ projects onto the image plane. This schematic relation constrains geometry. A changed camera should reveal a different view of the same point. Tracking through time is also necessary for moving points and their motion.

A plausible frame possibly shows the handle at a position that looks correct. A spatial representation must also agree with other views. For this reason, I think geometry will continue to be relevant as models scale. Each representation alone gives no drawer friction. Neither is sufficient evidence of stable gripper contact.

MASt3R and VGGT make learned geometry easier to reuse. They do not show that optimization disappeared. MASt3R’s pairwise geometry supplies input to a larger reconstruction pipeline. VGGT processes views together to predict cameras, depth, point maps and tracking features.

MASt3R’s matching-only ablation has lower pose accuracy than joint matching/3D supervision in its validation setup. I think geometry helps the model learn information that a correspondence objective misses in this example.

VGGT removes camera, depth or tracking supervision one at a time and measures point-map estimation. Joint prediction helps on that test. The paper gives test results for feature transfer for novel-view synthesis and dynamic point tracking. The paper gives large non-rigid deformation as a failure case. These results show that other tasks can use the learned geometric features.

For unified perception-action learning, I think methods to get 3D structure and reuse it must scale. Feed-forward estimation can help. A point map alone supplies no action labels, contact dynamics or control objective.

Predict how the scene changes

Researchers use “world model” for different goals. These include prediction of abstractions for reasoning and generation of detail for simulation and interaction. The source taxonomy shows those goals. I use four requirements to examine contributions to perception and action.

Video adds motion, occlusion, reappearance and interaction to static reconstruction. A large corpus includes temporal patterns that are not in a robot laboratory’s recorded scenes. Prediction is a learning task for changes in scene state from broad visual experience.

This notation shows that connection:

Here $b_t$ is a compact representation of visual history, and the model predicts the next observation. The model must keep object identity and geometry. It must then model the result of the agent’s selected intervention.

For the drawer, each source constrains a different error. If the camera returns, geometry constrains the handle’s expected position. For a drawer in motion, motion history helps predict how that motion continues. A simulator can show contacts and forces when scene and material assumptions are known. I expect systems to use these sources together according to the task.

VGGRPO connects diffusion latents to a geometry model. It then does post-training with camera-motion smoothness and geometric reprojection rewards. The ablation shows different effects on camera stabilization and repaired geometry. The rewards are for different errors. It also compares geometry backbones with their static/dynamic scene assumptions.

The result shows that a training reward can make generated geometry better. This is a claim about geometry, not learned physics. A learned geometry model supplies the reward. That reward does not verify forces or action results. I think physical grounding depends on the properties that training measures and on the scene representation.

Predict the consequence of an executable action

Passive video shows how scenes usually change. Passive video alone cannot identify outcomes of actions that are rare or not in the data. Smooth motion is not sufficient evidence of correct contact, inertia, collisions or object consistency.

The predictive interface changes from a continuation to a conditional result:

Here $a_t$ must represent an executable intervention: a gripper command, camera motion or a different defined action. A variable with the name “action” does not identify its causal effect. Data collection, action coverage, embodiment (body, sensors and action space) and hidden state limit the counterfactual claims the model can support.

A human video possibly shows a drawer that opens. 3D perception finds the handle. The robot must select an action for its body. Examples include wrist rotation, a changed grasp and a stop when resistance increases. Aligned observation-action trajectories connect commands to scene changes. New feedback shows when a prediction is incorrect.

I see video world models as a bridge to perception-action learning, not a complete solution. Video supplies broad temporal experience. Geometry constrains spatial state. Action-grounded data connects predicted results to the agent’s body. A deployed controller must decide how far to trust an approximation before it gets new observations and corrects errors.

Transfer tests of the ImageNet analogy

ImageNet combined scale, defined tasks, shared benchmarks and a stable optimization target. These conditions let researchers organize their work. Image labels give information about observations. Action data must align observations, interventions, time, bodies and outcomes. Video scale alone does not give an action-learning benchmark.

I recommend these transfer tests of the analogy:

1. Reserve new objects or environments in an action-grounded task family.
2. Measure if pretrained perception or dynamics decreases the number of interactions necessary for success.
3. Change the embodiment through a defined action mapping.
4. Give geometry, prediction error and task success as different quantities.

Better rendering cannot replace better control.

Feed-forward geometry makes perception easier to reuse. Geometry rewards can make video consistency better. General perception-action transfer stays untested in these results.

I think perception-action learning is approaching its ImageNet moment. Broad predictive pretraining can help with action if the research community supplies grounded interventions and evaluation measures the agent’s task performance. The research task is to make that condition concrete.

@article{ji2026on3dvideoworldmodelsandtheapproachingimagenetmomentofperceptionactionlearning,
  title = {On 3D, Video World Models, and the Approaching ImageNet Moment of Perception-Action Learning},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {March},
  url = {https://fusheng-ji.github.io/blog/posts/on-3d-video-world-models-and-the-approaching-imagenet-moment-of-perception-action-learning/}
}

References

1. Vincent Sitzmann. The flavor of the bitter lesson for computer vision. The essay examines the future of computer vision beyond intermediate tasks.

2. Xun Huang. Towards Video World Models. The blog examines video world models, causality, physical accuracy and interaction.

3. Jon Barron’s talk examines 3D vision in the age of video models. It also examines the possible obsolescence of 3D.

4. Saining Xie’s talk examines world models and related research directions.

5. Leroy et al., Grounding Image Matching in 3D with MASt3R. The paper gives geometry-grounded pairwise matching with joint matching and reconstruction supervision.

6. Wang et al., VGGT: Visual Geometry Grounded Transformer. The paper examines direct prediction of more than one 3D scene quantity from visual input.

7. VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward is the geometry-reward example in this article. Its result gives evidence of better geometric consistency.

Note: This blog was polished with the assistance of ChatGPT (GPT-5.4 Thinking).

Updated 03 November 2026