BLOG ILLUSTRATION / EXECUTION STORY

Scripted motion and camera controls

Directing the World generates blocks from scripted SMPL motion, camera paths and kept KV history.

Scripted motion and camera controlsDirecting the World generates blocks from scripted SMPL motion, camera paths and kept KV history. Denoising timestep τ SMPL scripts Each actor’s body control Dynamic projection timestep-conditioned Camera path prescribed trajectory Camera ControlNet Plücker-ray features Video history initial image + past blocks KV memory kept video state MMPL rollout next video block generated block → history
Author-redrawn Directing the World block diagram. Timestep-conditioned SMPL features, camera control and kept video history supply inputs for generation. Generated state supplies input for the next block. The method uses training for motion control before training for camera control. New controls use a higher learning rate. The video prior uses a lower rate or is frozen in the subsequent stage. This diagram does not show detailed frame plans or result images. Open full-size SVG