RoboAware: Learning to Coordinate Embodied Skills from Counterfactual Outcomes

arXiv:2610.11480 · cs.RO, cs.AI · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "RoboAware: Learning to Coordinate Embodied Skills from Counterfactual Outcomes".

Rosa: The gist The RoboAware framework learns state-dependent responsibility between modular composition and end-to-end control while keeping the coding agent and API library fixed.

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called RoboAware: Learning to Coordinate Embodied Skills from Counterfactual Outcomes, and what it claims is that you can learn state-dependent responsibility between modular composition and end-to-end control while keeping the coding agent and API library fixed.

Dev: It sounds like they're tackling a problem where you have these two different ways to build robot skills—modular parts versus these big end-to-end policies—and the challenge is deciding which one to use in any given situation.

Taro: The core idea seems to be that the system needs to anticipate which policy family will succeed based on the current physical state of the environment, not just pick one arbitrarily.

Rosa: Exactly, and they propose this P5 skill schema which defines those semantic boundaries where responsibility is assigned, and then they use counterfactual outcomes to learn a coordinator.

Dev: So how do they actually make this learning happen? It sounds like the methodology involves formulating it as a hierarchical MDP based on that P5 structure.

Taro: They decompose every skill into five stages—perceive, propose, pre-manipulate, perform, and post-manipulate—which gives them a unified way to look at both simple moves and complex handovers.

Rosa: And they use something called State-Locked Counterfactual Branching or SCB during training to make those policy family outcomes observable when they aren't naturally visible.

Dev: That sounds like they are using Execution-Aware Q-learning, or EAL, which combines Monte Carlo tree search with Q-learning to distill these outcomes into values based on the current state and the chosen skill family.

Taro: The learning target is trying to get a Q value that approximates the maximum expected return from a specific policy family given the state and the chosen skill.

Rosa: And then at test time, this learned value translates into a relative responsibility preference over admissible families, where they select the best one based on those learned values.

Dev: The coordinator picks c* i by maximizing that expression, which means it's choosing the policy family that the system has learned is best suited for whatever state it's in.

Taro: If you think about this from a real-world autonomy standpoint, this means the AI isn't just blindly following one path; it’s intelligently switching its strategy between composing modules and using a single end-to-end controller based on what the environment is doing right now.

Rosa: The results they show on one hundred tasks across three different benchmarks give them a SOTA seventy-seven point zero percent overall success rate, which is pretty solid compared to code-as-policy and VLA-harness baselines <ref:2610.11480#pg3,a SOTA 77.0% overall success rate>.

Dev: That seventy-seven percent figure is what the paper reports when evaluating the learned coordinator on those tasks, and they confirm that ablation results show this design works well because both families need to be coordinated by state <ref:2610.11480#pg3>.

Taro: What's interesting is how this relates to real-world deployment, Rosa; if we take these learned values and apply them, it suggests a level of coordination that might allow the system to handle unexpected physical shifts better than just sticking to one policy style.

Rosa: It definitely points toward a more robust system for embodied manipulation, but they also flag some things they don't fully solve yet.

Dev: Right, what are those limitations? The paper mentions that their current integration is limited to modular composition and end-to-end policies specifically, and future work involves extending it to generalizable reinforcement learning policies or retrieval-based policies.

Taro: And they also mention that dealing with finer-grained interruption when observations are ambiguous still seems like an open problem for this framework.

Rosa: It sounds like the paper confirms that while they’ve built a strong state-conditioned coordinator, the next steps involve making it more adaptable to completely different types of policies and handling those tricky moments when things aren't perfectly clear.

Dev: So, to sum up what we're hearing about RoboAware: Learning to Coordinate Embodied Skills from Counterfactual Outcomes, it’s a framework that learns which policy family—modular composition or end-to-end control—is the right choice at any given moment by looking at counterfactual outcomes.

Taro: It changes how we think about skill orchestration in embodied AI, moving away from just picking one method and toward dynamic coordination based on the physical situation.

Rosa: And for listeners, it means that as these agents get more complex, they won't just be running a single script; they’ll be making real-time decisions about how to combine different ways of doing things based on what the robot is actually sensing.

Conclusion: Rosa: So, we're wrapping up with some thoughts on RoboAware by the authors of that paper on learning to coordinate embodied skills from counterfactual outcomes.

Dev: It really focuses on how this system learns to switch between modular composition and end-to-end control based on what the robot is doing right now.

Taro: The core mechanism is this state-dependent responsibility learning, which means it figures out which policy family to use depending on the environment.

Rosa: It keeps the coding agent and that API library totally fixed while letting the AI figure out how to orchestrate those different skill styles.

Dev: They use a P5 skill schema to define those boundaries where this responsibility gets assigned, which is kind of a structured way to decompose skills into stages like perceiving and then performing actions.

Taro: And they train this by using counterfactual outcomes from simulations, which is the trick to observing those policy family results when you can't see them directly during training.

Rosa: The main result they show is that this approach reaches a seventy-seven percent overall success rate across a hundred different tasks, beating some existing code-as-policy and VLA harnesses.

Dev: But we have to remember the caveats, right? They confirm that both families need to be coordinated by state for those high success rates to show up.

Taro: What this means for someone just listening is that it’s not about one perfect way to build a skill; it’s about having a smart coordinator that knows when it needs a modular part versus when it needs a full end-to-end policy.

Rosa: The paper confirms they keep the core agent and API structure locked down, which is important for practical deployment outside of just the lab.

Dev: It’s about making the system robust enough to handle those shifts in responsibility during actual execution, not just in simulation.

Taro: And while it’s doing a lot with modular composition and end-to-end policies now, the authors admit they haven't fully extended it to general reinforcement learning policies yet.

Rosa: So the implication is that this framework gives us a concrete way to handle skill orchestration right now, but there’s still room to make it work for even more complex AI behaviors.

The Chinese University of Hong Kong · The Hong Kong University of Science and Technology · Knowin AI · Peking University · University of the Chinese Academy of Sciences

cs.RO, cs.AI

Submitted: 2026-10-08

Updated: 2026-10-08

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The gist The RoboAware framework learns state-dependent responsibility between modular composition and end-to-end control while keeping the coding agent and API library fixed.

Key concepts

P5 Skill Schema
This schema breaks down every skill into five distinct stages: perceive, propose, pre-manipulate, perform, and post-manipulate. It provides a universal way to represent any skill by defining the semantic boundaries where responsibility should be assigned.
State-Locked Counterfactual Branching (SCB)
SCB is a training technique used during learning. It allows the system to observe outcomes for different policy families by executing paired code blocks from identical restored states, making it possible to see which control strategy performs better in specific situations.
Execution-Aware Q-learning (EAL)
EAL combines Monte Carlo tree search (MCTS) with Q-learning. It is used to distill the outcomes observed via SCB into concrete, family-conditioned values. This process helps the system learn which policy family performs best given a specific state and action.
Relative Responsibility Preference
This is the mechanism used at deployment where learned values determine a preference between different control families. The coordinator selects the policy family that has the highest learned value for the current state, effectively choosing between modular composition and end-to-end control.

Terminology

Summary

The gist The RoboAware framework learns state-dependent responsibility between modular composition and end-to-end control while keeping the coding agent and API library fixed.

How it works

RoboAware formulates heterogeneous policy-family orchestration as stage-conditioned responsibility learning The P5 skill schema defines the semantic boundaries at which responsibility is assigned and SCB exposes both family outcomes at such a boundary by executing paired code blocks from identical restored states. This process involves selecting a policy family based on learned values, as the coordinator selects c∗i = arg maxc∈Ci ρθ(c xi).

Skill Preparation

The P5 skill schema decomposes each skill into five stages: perceive, propose, pre-manipulate, perform, and post-manipulate. This schema provides a unified representation for arbitrary skills, including primitive skills (e.g., pick, place, push, pull) and compositional skills (e.g., handover). An API library is curated around two complementary families: the Mod family uses an explicit intermediate interface for composability and controllability with end-to-end policies for contact-rich local execution.

Execution-Aware Learning

The EAL framework learns family-conditioned outcome values from controlled simulator experience. State-Locked Counterfactual Branching (SCB) is the training-time operator that makes the otherwise missing policy-family outcomes observable. Execution-Aware Q-learning (EAL) combines Monte Carlo tree search (MCTS) with Q-learning to distill these outcomes into family-conditioned values. The learning target is Qθ(xi, c) ≈ maxπ EπPTj=iγj−i r˜c i xi, ci = c.

Test-Time Skill Coordination

At deployment, the learned values induce a relative responsibility preference over admissible families: ρθ(c xi) = exp(Qθ(xi, c)) / Σexp(Qθ(xi, c′)). The RoboAware coordinator selects c∗i = arg maxc∈Ci ρθ(c xi) (equivalently arg maxc Qθ(xi, c); and the coding agent generates ai ∼ πcode(· xi, c⋆ i).

Evaluation and Results

Comprehensive single-episode evaluations on 100 tasks show that RoboAware reaches a 77.0% overall success rate. This performance outperforms existing code-as-policy and VLA-harness baselines. The framework demonstrates that both families must be coordinated by state to achieve high success rates, as shown by ablation results.

Limitations and Future Work

RoboAware currently integrates only modular composition and end-to-end policies. A future direction involves extending it to generalizable reinforcement learning policies, retrieval-based policies, and policies learned from demonstrations. Additionally, finer-grained interruption under ambiguous observations remains an open problem. The study also confirms that the small average SR gap across coding-agent backends and embodiments is attributed to P5, which limits each generation to one code block ai ∈ Aci(xi, zi, gi).

The RoboAware framework learns state-dependent responsibility between modular composition and end-to-end control while keeping the coding agent and API library fixed.

Improvements for AI systems

  1. A hierarchical MDP based on a P5 Skill Schema selects a policy family at each stage decision boundary to produce a code block; this allows for family-conditioned coordination by learning family-conditioned values Qθ(x, c) which encode a relative responsibility preference over modular composition and the end-to-end family.

  2. State-Locked Counterfactual Branching (SCB) restores the same training state to generate and execute a code block from each admissible family, exposing outcomes that selected-branch experience leaves unobserved, providing paired locked-state outcomes for learning.

  3. Execution-Aware Learning (EAL) uses Monte Carlo tree search (MCTS) guided by UCB selection and Q-learning to distill these counterfactual outcomes into family-conditioned values, allowing the coordinator to select the policy family according to observable context at deployment.

  4. The system can coordinate modular composition with end-to-end control by curating an API library that combines typed modular interfaces for composability and controllability with end-to-end policies for contact-rich local execution.

  5. The agent can generate complete executable code blocks where the selection of the policy family is conditioned on context, as seen in the deployment phase: the frozen coordinator selects the policy family according to observable context, and the frozen coding agent generates the next local code block.

Abstract

Embodied coding agents can combine modular robot skills with frozen end-to-end policies, yet effective composition requires anticipating which policy family will succeed in the current physical state. We present RoboAware, which builds on coding agents' skill orchestration by learning only a state-conditioned responsibility coordinator from counterfactual outcomes. Inspired by the success of REPL, we propose the P 5 schema and formulate a hierarchical MDP based on it. P 5 organizes skills uniformly into five semantic stages, defining where responsibility can be compared. To address the lack of counterfactual branch outcomes in existing work, we introduce State-Locked Counterfactual Branching (SCB), which restores the same training state to generate and execute a code block from each admissible family, exposing outcomes that selected-branch experience leaves unobserved. Building on this, we propose Execution-Aware Learning (EAL), which combines Monte Carlo tree search with Q-learning to distill these outcomes into family-conditioned values. At deployment, the coordinator selects the policy family according to observable context, and the frozen coding agent generates the next local code block. Comprehensive single-episode evaluations on 100 tasks show that RoboAware reaches a 77.0% overall success rate, with SOTA averages of 90.0% on RoboSuite, 73.8% on diverse LIBERO-Pro task clusters, and 90.0% on challenging RoboTwin bimanual tasks, outperforming existing code-as-policy and VLA-harness baselines.

Sources

Related papers