ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

summary

The gist

Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies The key insight is to shift adaptation from optimizing actions to…

In short

ExRL introduces Execution at the Right Length (ExRL) to adapt long-chunk policies efficiently. Instead of optimizing raw actions, it learns an execution-length policy that dictates how long each predicted action chunk should be executed. This shifts adaptation from complex action optimization to a simpler decision about duration, allowing sample-efficient adaptation across diverse tasks.

Key concepts

Execution at the Right Length (ExRL)
This is the core idea where an adapter predicts the optimal length for executing each predicted action chunk. It transforms policy adaptation from optimizing what to do (the action) to deciding how long to do it, keeping the adaptation space low-dimensional.
Execution-Length Policy ($\pi_{ex}$)
This is the learned component that takes the current state and the base policy's output and predicts a feasible execution length. It operates in a discrete space of possible lengths (K), allowing raw actions from the base policy to be executed for variable durations.
Duration-Aware TD Target ($y_t$)
The target signal used to train the critic is modified to account for the chosen execution length ($k_t$). This structure mirrors semiMarkov decision processes, ensuring the critic learns by comparing its prediction against rewards accumulated over variable time steps.
Sample-Efficient Adaptation
ExRL enables adaptation using minimal task-specific tuning and few samples. By optimizing the execution length rather than raw actions, it allows diverse frozen policies to adapt effectively in both simulation and real-world scenarios, often requiring very little physical interaction.

Terminology used across episodes

This episode discusses

The paper

ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control · Read on arXiv

Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao

UESTC University · Tongji University · The Hong Kong Polytechnic University

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control".

Paper summary: Rosa: Thinking about this paper, "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control", the core idea is making execution length a variable we optimize instead of just the raw action.

Dev: The authors show how this adaptation works by adding a lightweight execution length policy on top of an existing chunked policy and training it using off-policy RL methods to control when we stop executing a predicted chunk.

Taro: What I see is that they are making long horizon tasks more manageable for robots by giving them a control over their own temporal planning during execution.

Rosa: It means we can adapt complex, long action policies without needing massive amounts of task specific hyperparameter tuning for every single new environment.

Dev: The implication is that if you have a good base policy, ExRL lets you tune the *how long* parameter efficiently rather than retraining the whole action space from scratch.

Taro: It suggests a way to get sustained motion and local adjustments balanced across different manipulation tasks by learning when to stop executing based on what’s happening in the state.

Rosa: The paper points out that while they show success with frozen policies, further work could involve incorporating online behavior cloning to address failures when the base policy makes mistakes during real world interaction.

Dev: And they also flag that understanding how this adaptive replanning interacts with overlapping inference and execution is important for deploying this in systems where latency is a major concern.

Conclusion: Rosa: So we're wrapping up on ExRL, which is about how you can adapt those long action policies without needing a huge amount of task specific tuning for every new thing you want to do.

Dev: Exactly, and the paper focuses on this idea of controlling the execution length instead of just changing the raw actions themselves.

Taro: It sounds like they're giving these complex, long horizon tasks a way to manage their own timing during execution.

Rosa: Right, so they formalize it as learning a separate policy that decides how long each chunk should run, and they train that using off-policy reinforcement learning algorithms.

Dev: That means the adaptation space stays small even though the action chunks get longer, which is a smart way to keep things manageable for real time loops.

Taro: I'm interested in how this affects real world stuff; it shows a consistent preference for shorter executions when the robot is near its target, but longer ones when it’s just moving freely.

Rosa: That’s interesting because it suggests a learned strategy that balances local adjustments with sustained motion across different tasks and even different starting points.

Dev: But what about the safety side? The paper mentions they keep the base policy frozen for now, which means we still have to trust the actions it proposes if something goes really wrong in the real world.

Taro: That's a fair point; they also noted that because they haven't added online behavior cloning yet, the performance is still limited by what those initial frozen actions can actually do.

Rosa: True, so while this shows great potential for sample efficient adaptation in simulation and even some real world tests with just ten minutes of physical interaction, it’s not a plug and play solution yet.

Dev: It points toward the next steps, like incorporating online behavior cloning to help the system recover when the base policy makes a mistake during live interaction.

Rosa: And that leads us right into how this adaptive replanning interacts with things like latency sensitive deployments, which is where we need to keep an eye on next.

More episodes

← Home