BLOG ILLUSTRATION / EXECUTION STORY

VIA: target selection and local execution

RGB-D observations update a visual interface. An agent reads screenshots and uses MCP tools to set gripper waypoints. A local backend runs the waypoints. The observation-only scenario executes no waypoint.

VIA: target selection and local executionRGB-D observations update a visual interface. An agent reads screenshots and uses MCP tools to set gripper waypoints. A local backend runs the waypoints. The observation-only scenario executes no waypoint. READ AND REASON INTERACT WITH TOOLS LOCAL CONTROL Visual agent MCP tool calls Web interface views + gripper target Robot backend local controller screenshot MCP request RGB-D waypoint Point cloud + camera views Agent-selected targets, local motion Loop end: end_episode
The author made this interface mechanism diagram. Screenshots and MCP tools let the agent select a virtual gripper waypoint. The robot backend supplies local control. The example separates observation from execution. It does not include source screenshots, measured task results or direct joint control. Open full-size SVG