Robot-Use Agents: When Frontier Models Leave the Screen

In brief

In brief

Robot-use agents can orchestrate existing physical skills and improve them through verified trials. The credible near-term stack keeps semantic planning, motion generation, authorization, and fast local control distinct.

  • Agentic robotics spans runtime orchestration and policy improvement, but a VLM cannot reason past the physical capabilities of its tools.
  • Multimodality matters because it opens new task environments and produces grounded trajectories that can improve even text-centric reasoning.
  • Safe deployment still separates proposal, authorization, fast control, and hardware protection.

A frontier AI system controlling a dual-arm research robot through separate planning, verification, safety, and local-control layers. Figure 1. The interesting shift is not a direct wire from a frontier model to every motor. It is the growing stack of interfaces through which a general agent can perceive, program, verify, and safely use a robot. Generated with GPT Image.

One robot benchmark produced two numbers that look as if they came from different eras. In a September 2026 RoboCurve evaluation, GPT-6 Astra placed a red block in a bowl in 19 of 20 trials. On precise puzzle-piece insertion, it managed 2 of 20—the same as Claude Fable 5.1. Astra also used about 6.2 times fewer output tokens than Fable 5.1 and cost less than half as much per run.

The 95% result is real. So is the 10% result.

Together, they show why “a general AI can now control robots” is too blunt. The same model can look remarkably competent when perception, geometry, and tolerance line up, then stall at the final millimeters of contact-rich manipulation.

The interface matters. The models observed three camera views and proprioception, then requested absolute end-effector poses; automatic inverse kinematics produced joint commands. Every trial used the same bimanual YAM arms, instructions, 20-call budget, and 25% speed cap. This measures model-guided manipulation through a particular control contract, not a cloud model replacing robot control.

Two panels compare RoboCurve success counts for block-to-bowl and puzzle insertion across Fable 5, Fable 5.1, and Astra, with trial, interface, token, cost, inverse-kinematics, and speed-cap conditions.
Figure 2. RoboCurve's capability cliff: Astra reached 19/20 on block-to-bowl but only 2/20 on precise puzzle insertion. All results use 20 trials, Cartesian end-effector targets, automatic inverse kinematics, and a 25% speed cap. Source data: RoboCurve; author-created visualization.

The change is happening in the integration layer. Frontier agents are beginning to handle task decomposition, tool selection, program synthesis, progress checks, and recovery—even while motion and protection remain elsewhere in the stack. That alone may be enough to change robotics.

Research for this article is current through September 8, 2026.

A Robot as a Tool

Phillip Isola's short essay offers the cleanest framing: cameras and state become observations; navigation, grasping, and motion become tools; the agent becomes the robot's puppeteer. The word use shifts the focus away from a policy trained for one body. Through an interface, a general agent can attempt a task on an unfamiliar machine before that experience becomes a weight update.

That is different from an end-to-end VLA, which maps vision, language, and robot state into actions. A robot-use agent may call a VLA, but it can also call inverse kinematics, a motion planner, a perception service, or generated code. Its job is composition: decide what to invoke, inspect the result, and recover when the world disagrees.

The lineage is familiar. SayCan chose among grounded skills, Code as Policies composed robot programs, and Inner Monologue replanned from feedback. Today's agents can also read unfamiliar APIs, inspect images and traces, write and test code, and carry state across long tasks. Robotics is now testing how far those software abilities travel.

Three Clocks and the Adaptation Boundary

“An AI controls a robot” collapses three clocks. A multimodal agent may take seconds to infer intent, maintain task state, or decide whether to ask permission. A learned skill policy turns “grasp the mug” into poses or action chunks at roughly 10–100 Hz. Servoing and protection handle force, balance, collision response, and emergency braking at hundreds of hertz or more.

Timing dictates the architecture. Semantic reasoning can be remote and deliberative. Contact recovery needs physical data and a fast loop. Protection must survive network loss and model delay. The useful question is not “agent or VLA?” but where knowledge lives, and how quickly each layer can correct an error. RoboCurve's simple task improved while insertion barely moved because the tasks stress different clocks.

Three horizontal swimlanes compare seconds-scale semantic planning, 10-to-100-hertz learned skill control, and hundreds-of-hertz-to-kilohertz servo and protection loops, including inputs, outputs, typical mechanisms, and failure modes.
Figure 3. Generality, dexterity, and protection live on different clocks. A semantic agent can plan and authorize; a learned policy closes the sensorimotor loop; servo and protection mechanisms must remain fast and local. Author-created synthesis diagram.

The adaptation boundary has moved as well. SayCan selected fixed skills; Code as Policies generated programs; Inner Monologue revised plans from feedback. π₀ and later VLAs put more dexterity into learned action chunks. By 2026, agents compose tools and repair programs, physical prompts place demonstrations in context, and world-action models spend test-time compute on predicted outcomes. “No fine-tuning” now hides several distinct mechanisms.

Timeline from SayCan in 2022 through code generation, broad action policies, tool composition, physical prompting, and world-action-model test-time search in 2026, showing adaptation moving from predeployment weights toward runtime context and predicted futures.
Figure 4. The adaptation boundary has moved in stages: select skills, write programs, learn broader action priors, compose tools, prompt motion, and evaluate predicted futures. The sequence is conceptual rather than a claim that each route replaces the previous one. Author-created synthesis diagram.

Six Routes Toward General Robot Behavior

Six routes make different bets about what must be learned before deployment and what can be assembled at runtime. Production systems increasingly mix them.

1. End-to-end vision-language-action policies

A VLA maps images, language, and robot state to action tokens, waypoints, or continuous chunks. Broad multi-robot training supplies reusable physical priors; chunking amortizes inference and lets a local policy react faster than a cloud agent. The tradeoff is expensive, heterogeneous action data and an opaque failure boundary. VLAs work best near their training support; long-horizon planning, permissions, and novel tool use remain awkward.

2. Physical prompting and in-context imitation

Physical prompting places a time-aligned observation–action demonstration inside the model's context. It can show contact angle, order, and style that language specifies poorly, without starting a new training job. GEN-1.5 and Behavior Prompting Policy test this route. Its limits are cross-body alignment, accidental motion in the demonstration, and whether one trace covers the current scene.

3. World-action models

A world-action model proposes an action and predicts its effect. τ₀-WM jointly predicts visual latents and continuous action chunks, then samples and rectifies weak candidates at test time. It reports training on roughly 27,300 hours across robot and human sources. Prediction becomes an executable critic, but a plausible video can still be wrong about contact, friction, or hidden state. This route adds rehearsal, not proof. My longer world-model overview covers the prediction side.

Official τ₀-WM overview showing diverse training sources feeding a video-action model and action-conditioned video simulator, followed by proposal, evaluation, revision, and real-robot deployment.
Figure 5. τ₀-WM combines a Video Action Model with an Action-Conditioned Video Simulator so test-time computation can propose, evaluate, and revise action chunks before execution. Source: Zhou et al., τ₀-WM, Figure 1.

4. Hierarchical planners over learned skills

Here a foundation model chooses among motion plans, learned grasp policies, navigation stacks, and VLAs. The semantic planner composes while each skill closes its faster loop. Harness VLA and Pigey route between analytic primitives and frozen policies; GaP compiles mixed skills into a typed graph. Modularity makes permissions and failures inspectable, but someone must define the skill ontology, and no sequence can supply dexterity absent from its components.

5. Code as policy and self-improving skill libraries

A coding agent writes a program over perception, IK, planners, VLAs, and motion primitives, then repairs it from rollout traces. CaP-X benchmarks the pattern; Waddle and ASPIRE accumulate tested skills; ENPIRE changes training recipes and policies through physical experiments. Code is reviewable and replayable, but valid syntax says nothing about frame transforms, stale perception, or unsafe recovery. A physical test runner needs tighter authority than a shell.

6. Visual-interface and computer-use agents

A visual-interface agent sees camera views and state, calls a small action vocabulary, and replans from fresh observations. VIA uses a browser-based 3D interface; OpenETA exposes observe, mark_point, and move_to; RoboCurve exposes Cartesian targets backed by IK. Such systems can inherit gains in vision, coding, and tool use without retraining a task policy. Any benchmark must also name the bandwidth and competence hidden behind the interface.

Six-card map of end-to-end VLA, physical prompting, world-action models, hierarchical planners, code-as-policy systems, and visual-interface agents.
Figure 6. Six routes differ in where competence is stored and how a new task is specified. The rows are not a maturity ranking; production systems increasingly mix them. Author-created synthesis diagram.

Why the Hybrid Stack Keeps Reappearing

The strongest systems mix routes. A frontier agent interprets the request and chooses a tool; a VLA handles continuous manipulation; a geometric planner enforces exact constraints; a local controller turns the result into a high-frequency trajectory. This looks less like a temporary compromise than a scalable division of labor: the layers run at different clocks, learn from different data, and require different guarantees.

Two-lane comparison between an end-to-end VLA that maps observations to motor actions and a robot-use agent that calls typed tools through a deterministic safety gate and local controller.
Figure 7. A VLA puts learned action generation on the main path. A robot-use agent adds a slower semantic loop around a VLA, physical prompt, world-action model, or planner/IK tool, but deterministic authorization and high-frequency control remain local. Author-created synthesis diagram.

The execution contract matters more than the model boundary: what the agent can observe, which actions it may request, which parameters are legal, what proves success, and who can interrupt.

Vincent Sitzmann argues that computer-use ability may transfer because the loop has the same shape: observe state, choose a tool, inspect the result, revise. Program synthesis transfers too. A model that can read an unfamiliar library, run a test, and patch code already has useful habits for operating a robot API. Better visual reasoning, spatial grounding, and tool use can therefore improve robot behavior without a new task-specific policy.

Watching Astra operate simulators, Blender, and real robots changed how I frame an older argument: is multimodality itself necessary for intelligence? The argument may be aimed at the wrong bottleneck. Multimodality immediately unlocks more kinds of work. A model can edit a scene, inspect a render, move an object, test a controller, and observe the consequence. Each task yields a grounded trajectory of images, state, actions, code, errors, and outcomes—not just another document to predict.

Those trajectories can improve capabilities that look purely textual. A failed manipulation becomes a debugging trace; a successful Blender workflow becomes a reusable program; visual differences supervise planning and verification. The result can be distilled into language, code, memory, or policy updates. Whatever one believes about the internal form of intelligence, multimodal action creates training environments and feedback that text alone cannot supply. Even text-centric reasoning may increasingly learn from it.

VIA makes the computer-use analogy literal. An off-the-shelf agent uses screenshots and a browser-based 3D interface, reporting 96.7% success across three LIBERO-Goal tasks and 100% on rainbow assembly without robot-specific fine-tuning or privileged simulator state. Point-cloud and camera views, MCP tools, a virtual gripper target, and a controller supply the contact loop the agent itself does not learn. The interface is part of the result.

Official VIA overview showing a visual agent interacting through MCP tools with a web interface containing point-cloud, third-person, and wrist-camera views, selecting virtual gripper waypoints that a controller executes in the robot environment.
Figure 8. VIA turns robot operation into a visual computer-use loop. Its result depends on an engineered visual–action interface—point-cloud and camera views, MCP tools, virtual gripper waypoints, and a controller—not direct joint control. Source: Hu et al., VIA, Figure 1.

Anthropic's Claude Plays Robotics reaches the same conclusion across simulation, a real Unitree Go2, and manipulation. Models struggled with difficult joints directly and improved when writing controllers, steering pretrained policies, or using perception tools. CaP-X isolates the effect across twelve models, seven controlled tasks, and 100 trials per tier: higher-level abstractions, interaction, grounding, and more test-time compute all change performance.

Tools are legitimate system components, but their competence should not be credited to the foundation model. IK contributes geometry; a VLA contributes robot data; a planner contributes collision assumptions. The benchmark unit is model plus interface plus execution budget.

Whether the transfer is “emergent” remains unknown. Proprietary training mixtures may contain robot code, simulation, trajectories, manuals, or benchmark-adjacent data. “No task-specific fine-tuning” describes deployment, not the provenance of competence.

GEN-1.5: The Policy Learns in Context

Generalist's GEN-1.5 is a robot foundation model, not a computer-use agent, but it adapts at inference time. It accepts up to 30 seconds of video memory with language, proprioception, and other sensors, then emits trajectories at 100 Hz. A user can place a 3–12 second sensorimotor demonstration in that window while current observations roll forward. Generalist calls this physical prompting.

The prompt contains synchronized observations and actions, not merely a video of the desired outcome. It may come from handheld grippers or a robot rollout. The model must infer the intended motion, map it into the current scene, and correct deviations in closed loop. Language says what; the trace can show how.

Generalist reports 59% ±10% average success across ten simple, short-horizon tasks after one prompt and zero gradient updates. With about five minutes of task data—roughly fifty demonstrations—and ten gradient steps, the average rises to 83% ±9%. The comparison separates two mechanisms: in-context prompting changes the evidence available now; optional fine-tuning changes the weights.

Robot data remains central. Generalist says the model trained for more than eight months on a proprietary physical-interaction data engine. Broad pretraining may turn a new task from learning a skill from scratch into configuring behavior the model almost knows. The evaluation also falls short of establishing a universal one-shot learner: the company controls the model, embodiment, tasks, demonstrations, and success judgments, and says prompted skills remain more brittle than fine-tuned ones.

GEN-1.5 puts inference-time adaptation at a second layer. The semantic layer adapts through language, tools, memory, and traces; the sensorimotor layer adapts through action-bearing demonstrations and short gradient updates. A likely stack uses both: an agent selects the goal and teaching signal, a physically pretrained policy produces responsive motion, and separate systems verify and constrain it.

GEN-1.5 physical prompting loop from a 3-to-12-second sensorimotor demonstration through a 30-second rolling context, GEN-1.5, 100-hertz trajectories, and closed-loop robot execution, with a separate optional 1-to-10-gradient-step adaptation branch and two developer-reported evidence cards.
Figure 9. GEN-1.5 places a 3–12 second sensorimotor demonstration inside a 30-second rolling context and emits 100 Hz action trajectories. The 59% ±10% zero-gradient result covers ten simple short-horizon tasks; 83% ±9% uses about five minutes of data per task and ten gradient steps. These are Generalist's developer-reported evaluations and are not directly comparable with RoboCurve, VIA, or public VLA benchmarks. Author-created visualization.

Two Agentic Loops: Orchestrating Behavior and Improving the Policy

The recent systems do not form one leaderboard. They probe different points in the architecture.

Work Agent interface Robot-specific training Feedback and recovery Real hardware Main boundary
π₀ family Images, language, state → action chunks Multi-robot pretraining plus post-training Receding chunks, visual feedback Yes Substantial action data and embodiment integration
Claude Plays Robotics Direct commands, code, RL, or pretrained policy Depends on interface Repeated trials and strategy revision Yes Direct joint and humanoid control remain weak
GEN-1.5 Sensorimotor demonstrations in a 30-second context Extensive proprietary physical pretraining; none for a one-shot task 100 Hz closed-loop policy; some recovery reported Yes Company evaluation; 59% ±10% on ten simple, short-horizon tasks
Behavior Prompting Policy One demonstration as a behavior prompt Diverse prompt pretraining Closed-loop execution Yes Task diversity and embodiment alignment
τ₀-WM Action proposals, predicted futures, ranking ≈27,300 robot/human hours Sampling and rectification Yes, as reported Plausible video can be physically wrong
VIA Screenshots and browser-style 3D tools None for the agent Observe and adjust each step No in the reported benchmark suite Visual interface abstracts away part of real-world dynamics
CaP-X Programs over perception and control primitives Training-free agent; optional CaP-RL Structured traces, visual differencing, retries Yes Reliability depends strongly on designer abstractions
Waddle Programs, skills, VLAs, robot APIs No retraining between claimed adaptations Tests, repair, shared skills Yes Vendor stack and task suite
ASPIRE Programs over a traced execution engine Agent builds skills from experience Failure diagnosis, repair, validation, evolutionary search Initial transfer evidence Final success and transfer remain task-dependent
ENPIRE Coding agents manage reset, verification, rollout, and policy improvement Agent can change heuristics, training code, BC, or RL policies Automated physical trials and verified rewards across up to eight stations Yes The reported 99% is on selected tasks; bad resets or verifiers corrupt the research loop
GaP Graphs from 51 initial MORSL skills Agents refine graph and parameters Simulation, node checks, edge execution Yes New eight-task suite; large skill scaffold
OpenETA In one configuration: observe, mark_point, and move_to No task-specific policy or VLA in that configuration One world-changing call, fresh observation, trusted receipts, replay Interface integration only Pass@5 measures task coverage across seeds, not reliable episode success; no audited robot task rate
Harness VLA Frozen VLA beside analytic tools No VLA fine-tuning; memory guides use Re-ground, re-stage, retry No reported hardware benchmark Frozen-VLA support ceiling
Pigey VLM routes short subgoals to π₀.₅ or a TAMP grasp planner No new data or post-training for the frozen backends Visual verification, replanning, backend routing, recovery Yes Partial observability can produce false success; frontier-model calls add latency and cost
Gemini Robotics ER 2 Typed tools, search, VLA, robot APIs Embodied-reasoning post-training Video monitoring, replanning Yes Delegates motor execution
RoboCurve Camera views, state, and Cartesian pose commands through IK No disclosed robot fine-tuning in the evaluation Up to 20 agent calls Yes 95% on simple placement, 10% on precise insertion

“Agentic robotics” currently covers two distinct loops. The first adapts behavior during a task: fixed perception, planning, and motor systems are decomposed, routed, checked, and retried. The second adapts the robot system across trials: an agent changes a program, computation graph, training recipe, or policy from physical feedback. Orchestration changes the path through existing capabilities; policy improvement changes what will be available on the next attempt.

Orchestration can expose competence that the policy already had

OpenETA is the cleanest VLA-free example among the new papers. In its Codex configuration, the planner receives only three embodied operations: observe returns selected views, mark_point turns image selections into a 3D target, and move_to previews or executes a gripper motion. After any world-changing call, the interface requires fresh evidence. Perception, projection, motion, gripper behavior, and the simulator still do substantial work. “VLA-free” describes the action toolset, not an unassisted VLM driving motors.

The headline result needs two denominators. With GPT-5.6 Sol, the system solved 92 of 130 LIBERO tasks on the first ordered seed and covered 117 of 130 at Pass@5. Pass@5 counts a task if any of its first five seeds succeeds; it is not a 90% episode success rate. In a separate frozen evaluation over 40 tasks and ten seeds each, OpenETA recorded 56 successful episodes out of 400, with at least one success on 18 of 40 tasks. Its real-robot backend reached interface-level integration only, and no self-evolution candidate passed the promotion gate. The result is still meaningful, but narrower: a stronger planner used the same small interface more effectively; reliable repetition remained much harder.

VLM–VLA hybrids start from the complementary observation: analytic tools excel at staging, geometry, and explicit constraints, while learned policies earn their place on irregular grasps and contact-rich motion.

Harness VLA, from Chao Yu's group and collaborators, treats a frozen VLA as a retryable contact primitive while the outer agent handles re-grounding, navigation, transport, release, and re-staging. Its September 2 v4 reports gains of 38.6 percentage points on LIBERO-Pro and 27.1 points on RoboCasa365, plus 58.4% on RoboTwin clean-to-randomized transfer. These are separate benchmarks, not a pooled rate. With the VLA frozen, the gain measures orchestration and memory within its existing skill support.

Pigey, from Liane Galanti, Dhruv Shah, and Tri Dao, makes the controlled comparison sharper. A frontier VLM sends concrete subgoals to either a frozen π₀.₅ policy or a TAMP grasp planner, then verifies progress and recovers. The robot, cameras, scene, demonstrations, and policy weights stay fixed between direct and agentic conditions. Across six LIBERO-PRO perturbation suites, mean success rises from 12.8% to 53.3%. On 30 real-robot tasks, gains concentrate in reasoning-limited categories—world knowledge, conditionals, multi-step behavior, and recovery—while easy pick-and-place changes little. Frozen policies that were near zero on those reasoning-limited tasks reportedly exceeded 90% inside the loop.

The authors call this the orchestration gap. The term avoids two misleading explanations. The gain is not new dexterity—the motor weights did not change—and it is not free intelligence. Pigey relies on competent frozen backends, repeated vision calls, and a verifier vulnerable to occlusion. Seven tested reasoners showed the effect, but frontier-model latency and cost keep this loop out of high-speed control. Orchestration recovers capability stranded at the policy boundary.

A second loop turns the Agent into a robotics engineer

CaP-X establishes this second interpretation: the action is executable code, and the environment returns enough structure for repair. Waddle packages the pattern as a persistent system whose agents write control programs, call learned policies, inspect runs, and save validated routines. In one LEGO example, the agent also repeats trials, collects data, and trains an ACT policy. It has moved from choosing a skill to helping create one.

ASPIRE turns the loop into a continual-learning architecture. It records per-primitive multimodal traces, diagnoses failures, validates repairs, and stores successful fixes as reusable skills. Its accumulated library averaged 31% zero-shot success on held-out LIBERO-Pro Long tasks, versus 4% for the cited prior methods; simulation-discovered skills also reduced real-robot debugging effort. More important than the absolute rate is the compounding mechanism: each rollout can leave behind retrievable procedural knowledge.

ENPIRE, a collaboration involving NVIDIA, CMU, and UC Berkeley researchers, moves this research loop onto a physical fleet. Eight bimanual YAM stations repeatedly reset and verify a task, deploy a candidate, collect reward and logs, then let coding agents change heuristics, tool calls, behavior-cloning pipelines, or reinforcement-learning code. Selected tasks including PushT, organizing pins, and cutting a zip tie reached 99% success. That figure belongs to policies developed for those tasks inside a highly engineered autoresearch environment, not to general robot use.

ENPIRE's main contribution is the harness around that number. For pin insertion, increasing the fleet from one agent to eight reportedly cuts time to near-perfect success from more than 1.5 hours to roughly 40 minutes. Physical scaling, however, brings idle robots while agents read logs or write code, superlinear token cost, and the risk that a flawed reset or verifier turns efficient optimization into efficient damage. Outcome checks and safety boundaries define what the research agent is allowed to learn.

GaP takes a structured route: agents build, rehearse, and tune a computation graph from 51 initial MORSL skills before an edge interpreter runs it. Nodes may include analytic perception, planning, IK, control, and a VLA, so this is not “pure VLM.” Across four simulated and four hardware tasks, reported physical results include 25/25 grocery-order, 28/30 packing, and 18/20 popcorn trials. The custom suite, large skill library, and internal simulation are both part of the result and part of its reproduction burden.

Google DeepMind's Gemini Robotics ER 2 makes the hybrid boundary explicit. Its embodied-reasoning model consumes video, audio, text, and images, orchestrates tools, monitors progress, and hands motor execution to a lower-level VLA or robot API. It can reason about the next step while the current action runs. The systems above add another possibility: the outer agent can collect verified experience and improve the tool it will call next time.

Figure 10 compares these placements as system paths: where adaptation happens, where feedback returns, and where physical competence remains.

Three horizontal paths compare an agent or VLM using analytic tools, an agent improving programs and policies across verified trials, and an agent or VLM orchestrating a frozen VLA with analytic skills. Each path identifies the changing artifact, feedback loop, physical competence, and principal boundary.
Figure 10. Three places to put the Agent: over analytic tools, around a policy-improvement loop, or above a frozen VLA and analytic skills. The paths can be combined. “VLA-free” describes a particular execution toolset; it does not remove perception, geometry, inverse kinematics, planning, or local control. Author-created synthesis diagram.

Together, these systems show that general agents can contribute useful robot behavior without learning every task end to end. Robot-specific learning remains essential.

The progress is more specific than “models got smarter.” Physical priors are broader: multi-robot and human-interaction data reduce how much motion each new task must teach. Action representations are more forgiving: continuous chunks preserve coordination across joints, while Cartesian poses and typed skills trade bandwidth for easier reasoning. Both gains still depend on embodiment integration and contact coverage.

Test-time compute has entered the physical loop. A system can sample trajectories, predict futures, inspect a visual difference, call another tool, or reason again after failure. CaP-X and τ₀-WM do this within an attempt; ENPIRE and GaP extend it across trials. The cost is harsher than in language: search delays action, while physical experiments consume wear, energy, resets, supervision, and safety budget.

Feedback is becoming an interface rather than an afterthought. “The command ran” does not mean the task succeeded. Images, controller state, contact signals, structured errors, and outcome receipts locate failure. An IK rejection should not cause the planner to rewrite the goal; an ambiguous goal should not make a VLA retry forever. Priors improve proposals, predictive models rank them, agents compose them, and traces diagnose the result.

The evidence remains uneven. Success rates depend on resets, initial-state variation, intervention, speed limits, termination rules, and whether partial progress counts. Evaluations should state what is genuinely held out—embodiment, API, object distribution, physical skill, or composition—and publish prompts, traces, controller contracts, failure videos, cost, and intervention rules. Cross-layer orchestration and inference-time adaptation are clearly improving; absolute reliability remains specific to the task, interface, and execution budget.

Risk, Safety, and the Execution Boundary

The computer-use analogy becomes dangerous when it hides the asymmetries. A failed click can usually be undone; a failed grasp can drop glass, deform a part, spill liquid, or move the scene outside the camera's view. Trial and error needs a recovery environment, not merely permission to continue.

The physical world also has no universal accessibility tree. A robot receives incomplete images, noisy depth, occlusion, calibration error, and platform-specific proprioception. Its structured state has to be constructed—and may already be stale when an action begins.

Physical authority also differs from digital access. A robot agent can leak data or buy the wrong item, but it can also damage property or injure a person. Prompt injection, compromised sensors, a hallucinated affordance, or stale state can cross into motion. Exposing an actuator API is therefore an authorization decision, not a convenience.

Robot evaluations usually report task success. Deployment depends just as much on how failures happen. A system that succeeds 95% of the time and stops safely on the rest is a different product from one that succeeds 99% but occasionally applies force in the wrong place.

The relevant risks enter at every layer:

Failure surface Typical mechanism Physical consequence Engineering response
Observation and state Occlusion, calibration error, stale frames Action targets a scene that no longer exists Timestamped state, multi-view and contact checks
Identity, goal, and security Ambiguity, wrong principal, prompt injection A valid skill serves the wrong goal or user Authentication, provenance, capability scopes, confirmation
Planning and learned motion Bad affordance, world-model error, OOD action Collision, spill, excessive force Typed skills, critics, receding horizons, action envelopes
Timing and control Cloud delay, stale command, dropped packet Late correction or uncontrolled continuation Local watchdog, expiry, force limits, hardware stop
Verification and learning False success, bad reset, unsafe memory Wrong reward or propagated failure Independent checks, retry budgets, promotion gates, rollback
Fleet and operations Correlated trials, policy spread, hidden intervention, surveillance Wear, unsafe rollout, privacy loss, false confidence Canaries, signed artifacts, budgets, retention rules, public traces

The same properties that make agents useful amplify risk. A model that reads the web, writes code, and calls robot tools can connect a malicious label, QR code, skill description, or memory entry to an actuator. Provenance and identity must travel with data, and robot capability scopes should be narrower than the agent's information access.

Reuse creates correlated failure. One frame-convention bug or unsafe recovery can spread through a skill library or fleet. ENPIRE's reset loop, ASPIRE's memory, and GaP's graph rehearsal make the verifier part of the control system: a false success can promote a brittle policy. Discovery therefore needs independent reproduction, non-regression tests, signed artifacts, staged rollout, and rollback.

Persistent perception adds privacy and accountability risks. Video, audio, maps, identities, and physical demonstrations reveal more than the immediate task, while a modular incident spans the agent, planner, vendor policy, and controller. Retention limits, access logs, visible recording state, and an end-to-end decision trace are architectural requirements.

Capability and authority therefore need separate measurements. As the planner becomes more general, the execution boundary must become more explicit. Every transformation—from input to plan, request, policy output, and controller command—needs an interception point independent of the model that proposed the action.

Horizontal execution pipeline from external input through semantic planner, tool request, learned policy, local controller, and robot, with amber risk entry points and green local interceptors for provenance, state freshness, typed preconditions, distribution and action envelopes, watchdogs, force limits, and hardware stops.
Figure 11. Risk cannot be filtered only at the model entrance. Provenance, state freshness, typed preconditions, action envelopes, watchdogs, force limits, and hardware stops belong at the execution boundaries where errors can still be contained. Author-created synthesis diagram.

Instructions can guide an agent toward safer choices; they cannot make unsafe motion impossible.

Google DeepMind's ASIMOV-Agentic evaluates whether a reasoning agent refuses actions that violate operational constraints, triggers protective stops during faults or unsafe human proximity, shields a VLA from infeasible or out-of-distribution requests, and asks for help when instructions or scenes are ambiguous. These semantic capabilities sit above the mechanisms that enforce speed, force, workspace, and collision limits.

Physical Agentic AI provides a sharper architectural result. Its foundation-model planner proposes robot-skill assignments, but a deterministic orchestrator validates one dispatch at a time against capabilities, state, and workflow rules. Retrieval raised skill grounding from 51% to 96%, yet informed planners still attempted faulted actions in 23–29% of steps under the study's injected-fault conditions. Per-dispatch enforcement reduced false dispatches to zero without false blocks in that evaluation.

One gate does not solve robot safety. The result instead separates planning quality from authorization: a better model lowers the rate of bad proposals; a deterministic boundary decides whether a proposal may become motion.

A credible robot-use stack should therefore provide:

  • typed, capability-scoped tools rather than arbitrary actuator access;
  • fresh state plus local speed, force, workspace, and proximity limits;
  • explicit outcome evidence and bounded retries after each skill;
  • an independent interrupt path, watchdog, and hardware stop;
  • traceable decisions and human takeover outside the allowed envelope.

Safety is not a paragraph in the system prompt. It is a property of the path from proposal to actuation.

The New Scaling Unit

Robot learning has usually scaled through data and policies: collect more demonstrations, train a broader model, add embodiments, and fine-tune for deployment. That recipe will continue. A strong VLA can compress physical experience into a responsive local policy, and the human-to-robot interfaces used to create such data are improving quickly.

Robot-use agents add another scaling unit: the skill contract.

A useful contract states what a skill does, which observations and parameters it accepts, which preconditions must hold, what marks completion, which failures it reports, and which safety envelope contains it. A general agent can then discover, compose, test, repair, and reuse the skill. A VLA, world model, grasp planner, calibration routine, or teleoperator can all sit behind such a contract.

Physical prompting adds a second unit: the teaching trace. Alongside code or weights, a deployment can ship a short, provenance-checked demonstration that configures a broad policy in context. Reusable traces can capture what language specifies poorly—contact sequence, grasp style, or bimanual timing—while typed contracts carry permissions and completion criteria. Code, demonstrations, and weights become complementary forms of robot software.

This changes where engineering effort compounds. A better controller improves every task that calls it; a better perception tool improves every agent that consumes its state; a validated recovery routine enters a shared library. A stronger frontier model can attempt new compositions without recollecting demonstrations for every sequence.

The bottleneck moves rather than disappears. We need robot interfaces as disciplined as good software APIs, evaluation harnesses that expose physical failures, and safety systems that assume the semantic planner will sometimes be wrong. Hardware, robot data, and contact-rich policies become the substrate a more general agent can use.

The RoboCurve pair—19/20 for block placement, 2/20 for insertion—marks a moving integration boundary, not solved general robotics. Capability can arrive through a better frontier agent, a typed skill, a prompted local policy, or verified experience fed back into code and weights. If the interfaces become portable, behavior may spread faster than the hardware fleet changes.

The decisive artifact is not a demo of a model moving an arm. It is the execution contract around that motion: what the agent could observe, which tools it could call, what happened when it was wrong, and which layer had authority to stop it.

The robot may become another tool for a general agent. The hard part is making it a tool we can trust.

Citation

Please cite this article as:

Ji, Wenbo. “Robot-Use Agents: When Frontier Models Leave the Screen”. fusheng-ji.github.io (September 2026). https://fusheng-ji.github.io/blog/posts/robot-use-agents/

Or use the BibTeX entry:

@article{ji2026robotuseagents,
  title = {Robot-Use Agents: When Frontier Models Leave the Screen},
  author = {Ji, Wenbo},
  journal = {fusheng-ji.github.io},
  year = {2026},
  month = {September},
  url = {https://fusheng-ji.github.io/blog/posts/robot-use-agents/}
}

References

  1. Phillip Isola, Robot-use agents, September 7, 2026.
  2. Jay Chooi, GPT-6 Astra robot-control benchmark thread, September 5, 2026.
  3. RoboCurve, GPT-6 Astra on robotic manipulation, September 4, 2026.
  4. Vincent Sitzmann, comments on computer-use transfer, training data, and control speed, September 7–8, 2026.
  5. Anthropic, Claude Plays Robotics, July 9, 2026.
  6. Hu et al., VIA: Visual Interface Agent for Robot Control, 2026.
  7. Fu et al., CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation, 2026.
  8. Lu et al., ASPIRE: Agentic Skills Discovery for Robotics, 2026.
  9. Waddle Labs, Introducing Waddle: Agents that Control Robots, July 2026.
  10. Lima et al., Agentic AI for Robot Control: Flexible but still Fragile, 2026.
  11. Google DeepMind, Gemini Robotics ER 2 and ASIMOV-Agentic, 2026.
  12. Liu et al., Physical Agentic AI: An Architecture for Orchestrating a Robot Crew with LLMs, 2026.
  13. Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, 2022.
  14. Liang et al., Code as Policies: Language Model Programs for Embodied Control, 2022/2023.
  15. Huang et al., Inner Monologue: Embodied Reasoning through Planning with Language Models, 2022.
  16. Generalist Team, GEN-1.5: Embodied Foundation Models are One-Shot Learners, August 19, 2026.
  17. Black et al., π₀: A Vision-Language-Action Flow Model for General Robot Control, 2024.
  18. Patel et al., Behavior Prompting Policy: Demonstrations as Prompts for Manipulation, 2026.
  19. Zhou et al., τ₀-WM: A Unified Video-Action World Model for Robotic Manipulation, 2026.
  20. Xiao et al., ENPIRE: Agentic Robot Policy Self-Improvement in the Real World, 2026.
  21. Chen et al., GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness for Variational Automation Tasks, 2026.
  22. Chen et al., ETA: A New Agentic Paradigm for Embodied Tasks, 2026.
  23. Zhang et al., Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents, 2026.
  24. Galanti et al., Addressing the Orchestration Gap in Generalist Robots via Physical Agency, 2026.

Note: This blog was written and polished with the assistance of Codex (GPT-5.6 Sol).

Related notes