BLOG ILLUSTRATION / EXECUTION STORY

Genie action codes from video

Video tokenization and latent-action inference are parallel training branches into a video dynamics predictor. In the interaction trace, a selected learned action code replaces inference from video transitions. These codes are not calibrated robot commands.

Genie action codes from videoVideo tokenization and latent-action inference are parallel training branches into a video dynamics predictor. In the interaction trace, a selected learned action code replaces inference from video transitions. These codes are not calibrated robot commands. Video frames Unlabeled clips Tokenization Video tokenizer Video tokens Visual history Action inference Video transitions Action codes Discrete IDs Dynamics Token model Next tokens Video prediction with a learned action vocabulary Robot grounding necessary
Genie trains action-conditioned dynamics with video tokens and latent actions inferred from unlabeled transitions. These latent actions are not calibrated robot commands. Author-redrawn mechanism from Bruce et al., without game footage and generated results. Source: Bruce et al. Playback time is illustrative and is not measured runtime. Open full-size SVG