BLOG ILLUSTRATION / EXECUTION STORY

HumanEgo: predict actions from hand–object relations

A shared encoder and flow-matching policy use RGB observations and entity-relative hand tokens. The policy generates bimanual actions. Different training-only heads predict object motion, 2D traces and future tokens.

HumanEgo: predict actions from hand–object relationsA shared encoder and flow-matching policy use RGB observations and entity-relative hand tokens. The policy generates bimanual actions. Different training-only heads predict object motion, 2D traces and future tokens. RGB observation Visual encoder Shared context RGB + interaction state Tracked entities hands + objects Entity tokens object-relative hands Flow-matching policy bimanual action horizon Object motion 2D traces Future ICT FUTURE PREDICTION · TRAINING ONLY
The author made this diagram from the source figure. Entity tokens contain hand poses in entity frames. The action generator uses RGB and relational context. Dashed auxiliary heads supply training supervision only. The diagram does not show source photographs, prediction plots or task-success measurements. Open full-size SVG