ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control
summary
The gist
Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies The key insight is to shift adaptation from optimizing actions to…
In short
ExRL introduces Execution at the Right Length (ExRL) to adapt long-chunk policies efficiently. Instead of optimizing raw actions, it learns an execution-length policy that dictates how long each predicted action chunk should be executed. This shifts adaptation from complex action optimization to a simpler decision about duration, allowing sample-efficient adaptation across diverse tasks.
Key concepts
- Execution at the Right Length (ExRL)
- This is the core idea where an adapter predicts the optimal length for executing each predicted action chunk. It transforms policy adaptation from optimizing what to do (the action) to deciding how long to do it, keeping the adaptation space low-dimensional.
- Execution-Length Policy ($\pi_{ex}$)
- This is the learned component that takes the current state and the base policy's output and predicts a feasible execution length. It operates in a discrete space of possible lengths (K), allowing raw actions from the base policy to be executed for variable durations.
- Duration-Aware TD Target ($y_t$)
- The target signal used to train the critic is modified to account for the chosen execution length ($k_t$). This structure mirrors semiMarkov decision processes, ensuring the critic learns by comparing its prediction against rewards accumulated over variable time steps.
- Sample-Efficient Adaptation
- ExRL enables adaptation using minimal task-specific tuning and few samples. By optimizing the execution length rather than raw actions, it allows diverse frozen policies to adapt effectively in both simulation and real-world scenarios, often requiring very little physical interaction.
Terminology used across episodes
This episode discusses
- ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control · Paper Radio
- pi 0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
- pi* 0.6: a VLA That Learns From Experience
- Residual Off-Policy RL for Finetuning Behavior Cloning Policies
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- pi 0: A Vision-Language-Action Flow Model for General Robot Control
- pi 0.5: a Vision-Language-Action Model with Open-World Generalization
- RT-1: Robotics Transformer for Real-World Control at Scale
- pi RL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
- ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network · Paper Radio
- Soft Actor-Critic for Discrete Action Settings
- EXPO: Stable Reinforcement Learning with Expressive Policies
- What Matters for Batch Online Reinforcement Learning in Robotics?
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- OpenAI o1 System Card
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- OpenVLA: An Open-Source Vision-Language-Action Model
- Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement · Paper Radio
- Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?
The paper
ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control · Read on arXiv
Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao
UESTC University · Tongji University · The Hong Kong Polytechnic University
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control".
Paper summary: Rosa: Thinking about this paper, "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control", the core idea is making execution length a variable we optimize instead of just the raw action.
Dev: The authors show how this adaptation works by adding a lightweight execution length policy on top of an existing chunked policy and training it using off-policy RL methods to control when we stop executing a predicted chunk.
Taro: What I see is that they are making long horizon tasks more manageable for robots by giving them a control over their own temporal planning during execution.
Rosa: It means we can adapt complex, long action policies without needing massive amounts of task specific hyperparameter tuning for every single new environment.
Dev: The implication is that if you have a good base policy, ExRL lets you tune the *how long* parameter efficiently rather than retraining the whole action space from scratch.
Taro: It suggests a way to get sustained motion and local adjustments balanced across different manipulation tasks by learning when to stop executing based on what’s happening in the state.
Rosa: The paper points out that while they show success with frozen policies, further work could involve incorporating online behavior cloning to address failures when the base policy makes mistakes during real world interaction.
Dev: And they also flag that understanding how this adaptive replanning interacts with overlapping inference and execution is important for deploying this in systems where latency is a major concern.
Conclusion: Rosa: So we're wrapping up on ExRL, which is about how you can adapt those long action policies without needing a huge amount of task specific tuning for every new thing you want to do.
Dev: Exactly, and the paper focuses on this idea of controlling the execution length instead of just changing the raw actions themselves.
Taro: It sounds like they're giving these complex, long horizon tasks a way to manage their own timing during execution.
Rosa: Right, so they formalize it as learning a separate policy that decides how long each chunk should run, and they train that using off-policy reinforcement learning algorithms.
Dev: That means the adaptation space stays small even though the action chunks get longer, which is a smart way to keep things manageable for real time loops.
Taro: I'm interested in how this affects real world stuff; it shows a consistent preference for shorter executions when the robot is near its target, but longer ones when it’s just moving freely.
Rosa: That’s interesting because it suggests a learned strategy that balances local adjustments with sustained motion across different tasks and even different starting points.
Dev: But what about the safety side? The paper mentions they keep the base policy frozen for now, which means we still have to trust the actions it proposes if something goes really wrong in the real world.
Taro: That's a fair point; they also noted that because they haven't added online behavior cloning yet, the performance is still limited by what those initial frozen actions can actually do.
Rosa: True, so while this shows great potential for sample efficient adaptation in simulation and even some real world tests with just ten minutes of physical interaction, it’s not a plug and play solution yet.
Dev: It points toward the next steps, like incorporating online behavior cloning to help the system recover when the base policy makes a mistake during live interaction.
Rosa: And that leads us right into how this adaptive replanning interacts with things like latency sensitive deployments, which is where we need to keep an eye on next.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies