ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control".
Paper summary: Rosa: Thinking about this paper, "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control", the core idea is making execution length a variable we optimize instead of just the raw action.
Dev: The authors show how this adaptation works by adding a lightweight execution length policy on top of an existing chunked policy and training it using off-policy RL methods to control when we stop executing a predicted chunk.
Taro: What I see is that they are making long horizon tasks more manageable for robots by giving them a control over their own temporal planning during execution.
Rosa: It means we can adapt complex, long action policies without needing massive amounts of task specific hyperparameter tuning for every single new environment.
Dev: The implication is that if you have a good base policy, ExRL lets you tune the *how long* parameter efficiently rather than retraining the whole action space from scratch.
Taro: It suggests a way to get sustained motion and local adjustments balanced across different manipulation tasks by learning when to stop executing based on what’s happening in the state.
Rosa: The paper points out that while they show success with frozen policies, further work could involve incorporating online behavior cloning to address failures when the base policy makes mistakes during real world interaction.
Dev: And they also flag that understanding how this adaptive replanning interacts with overlapping inference and execution is important for deploying this in systems where latency is a major concern.
Conclusion: Rosa: So we're wrapping up on ExRL, which is about how you can adapt those long action policies without needing a huge amount of task specific tuning for every new thing you want to do.
Dev: Exactly, and the paper focuses on this idea of controlling the execution length instead of just changing the raw actions themselves.
Taro: It sounds like they're giving these complex, long horizon tasks a way to manage their own timing during execution.
Rosa: Right, so they formalize it as learning a separate policy that decides how long each chunk should run, and they train that using off-policy reinforcement learning algorithms.
Dev: That means the adaptation space stays small even though the action chunks get longer, which is a smart way to keep things manageable for real time loops.
Taro: I'm interested in how this affects real world stuff; it shows a consistent preference for shorter executions when the robot is near its target, but longer ones when it’s just moving freely.
Rosa: That’s interesting because it suggests a learned strategy that balances local adjustments with sustained motion across different tasks and even different starting points.
Dev: But what about the safety side? The paper mentions they keep the base policy frozen for now, which means we still have to trust the actions it proposes if something goes really wrong in the real world.
Taro: That's a fair point; they also noted that because they haven't added online behavior cloning yet, the performance is still limited by what those initial frozen actions can actually do.
Rosa: True, so while this shows great potential for sample efficient adaptation in simulation and even some real world tests with just ten minutes of physical interaction, it’s not a plug and play solution yet.
Dev: It points toward the next steps, like incorporating online behavior cloning to help the system recover when the base policy makes a mistake during live interaction.
Rosa: And that leads us right into how this adaptive replanning interacts with things like latency sensitive deployments, which is where we need to keep an eye on next.
Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao
UESTC University · Tongji University · The Hong Kong Polytechnic University
cs.RO
Submitted: 2026-11-10
Updated: 2026-11-10
Project page: https://exrl-project.github.io
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 88/100
The gist: Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies The key insight is to shift adaptation from optimizing actions to…
Key concepts
- Execution at the Right Length (ExRL)
- This is the core idea where an adapter predicts the optimal length for executing each predicted action chunk. It transforms policy adaptation from optimizing what to do (the action) to deciding how long to do it, keeping the adaptation space low-dimensional.
- Execution-Length Policy ($\pi_{ex}$)
- This is the learned component that takes the current state and the base policy's output and predicts a feasible execution length. It operates in a discrete space of possible lengths (K), allowing raw actions from the base policy to be executed for variable durations.
- Duration-Aware TD Target ($y_t$)
- The target signal used to train the critic is modified to account for the chosen execution length ($k_t$). This structure mirrors semiMarkov decision processes, ensuring the critic learns by comparing its prediction against rewards accumulated over variable time steps.
- Sample-Efficient Adaptation
- ExRL enables adaptation using minimal task-specific tuning and few samples. By optimizing the execution length rather than raw actions, it allows diverse frozen policies to adapt effectively in both simulation and real-world scenarios, often requiring very little physical interaction.
Terminology
Summary
Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies The key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning. p. 0
The Core Concept
ExRL introduces Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. This method shifts adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. p. 0 The adaptation space remains low-dimensional while its expressivity grows with the chunk size. p. 1
Formalization of Adaptation
The method formalizes this idea as reinforcement learning over execution-length decisions. Given a pretrained action-chunked policy πbase with chunk size H, they learn an additional execution-length policy πex: S × AH → ∆(K), where K = K = the set of feasible execution lengths. p. 2 This allows the raw primitive actions to still be produced by πbase, while the learned execution-length policy acts in the much smaller discrete space K. p. 2
Training Signal Construction
The duration-aware TD target is defined as yt = kXt−1i=0γi r(st+i, at+i) + γktV (s′, a′), where kt is the chosen execution length. This target mirrors the Bellman backup used in semiMarkov decision processes (SMDPs). p. 2 The critic learns by minimizing the squared TD error Lq(ϕ) = Eh(Qϕ(st, at:t+H, kt) − yt)2. p. 2
Performance and Learning Patterns
ExRL enables sample-efficient adaptation of diverse frozen policies across simulation and real-world tasks. In real-world experiments, ExRL-Q approaches 100% success on stack cube with only ten minutes of physical interaction. p. 6 The learned execution length patterns show a consistent preference for shorter executions near the object or target than during free motion across tasks and seeds. p. 6
Policy Evolution and Benefits
ExRL can also leverage offline data by treating each H-step trajectory segment as a virtual action chunk, whose prefixes provide replay experience for different execution lengths. Successful online rollouts can be distilled back into the base policy using behavior cloning, improving its chunk predictions. p. 4 The adaptive strategy balances local adjustments and sustained motion to achieve optimal results across different manipulation tasks. p. 1
Limitations and Extensions
The real-world experiments keep the base policy frozen and do not yet incorporate online BC, meaning performance remains constrained by the actions the base policy can propose. Further extensions could involve incorporating online BC to address remaining failures in real-world tasks. p. 6 The work also notes that understanding how adaptive replanning interacts with overlapping inference and execution is an important direction for extending ExRL to latency-sensitive deployments. p. 7
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Sample efficient adaptation of long-chunk policies: ExRL enables sample-efficient adaptation of diverse frozen policies across simulation and real-world tasks with minimal task-specific hyperparameter tuning.
-
Policy agnostic interface for adaptation: ExRL adds a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms, allowing it to be applied to virtually any class of action-chunked policies without modifying the base policy itself.
-
Duration aware Q learning: The system learns a critic Qphi(st, at:t+H, k) that estimates the return obtained by executing the first k actions of a predicted chunk and then continuing with the current execution-length strategy, leading to a duration aware TD target.
-
Leveraging offline data for adaptation: ExRL can leverage general offline trajectories by reinterpreting arbitrary trajectory segments as off-policy experience under different execution lengths, further improving online sample efficiency.
-
Behavior shaping through learned patterns: ExRL learns that
across Lift, Can, Square, and real-world insertion, ExRL selects shorter executions near the object or target than during free motion,
concentrating replanning near interaction to balance local adjustments and sustained motion. -
Policy refinement via demonstration augmentation: Combining ExRL-Q with Online BC allows for rapid adaptation on tasks like ACT Can by distilling successful online rollouts back into the base policy, reaching near-perfect success competitive with the strongest demonstration based baselines.
-
Enhanced precision in manipulation: ExRL enables real-world adaptation where it approaches 100% success on stack cube with only ten minutes of physical interaction, and nearly 90% on long horizon clothing folding within 85 minutes, significantly outperforming fixed length execution across all tested tasks.
Sources
- ${\pi}_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- Residual Off-Policy RL for Finetuning Behavior Cloning Policies
- GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
- RT-1: Robotics Transformer for Real-World Control at Scale
- $\pi_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
- ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
- Soft Actor-Critic for Discrete Action Settings
- EXPO: Stable Reinforcement Learning with Expressive Policies
- What Matters for Batch Online Reinforcement Learning in Robotics?
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- OpenAI o1 System Card
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- OpenVLA: An Open-Source Vision-Language-Action Model
- Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
- Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement
- Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation