ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

arXiv:2611.13482 · cs.RO · Submitted 2026-11-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control".

Paper summary: Rosa: Thinking about this paper, "ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control", the core idea is making execution length a variable we optimize instead of just the raw action.

Dev: The authors show how this adaptation works by adding a lightweight execution length policy on top of an existing chunked policy and training it using off-policy RL methods to control when we stop executing a predicted chunk.

Taro: What I see is that they are making long horizon tasks more manageable for robots by giving them a control over their own temporal planning during execution.

Rosa: It means we can adapt complex, long action policies without needing massive amounts of task specific hyperparameter tuning for every single new environment.

Dev: The implication is that if you have a good base policy, ExRL lets you tune the *how long* parameter efficiently rather than retraining the whole action space from scratch.

Taro: It suggests a way to get sustained motion and local adjustments balanced across different manipulation tasks by learning when to stop executing based on what’s happening in the state.

Rosa: The paper points out that while they show success with frozen policies, further work could involve incorporating online behavior cloning to address failures when the base policy makes mistakes during real world interaction.

Dev: And they also flag that understanding how this adaptive replanning interacts with overlapping inference and execution is important for deploying this in systems where latency is a major concern.

Conclusion: Rosa: So we're wrapping up on ExRL, which is about how you can adapt those long action policies without needing a huge amount of task specific tuning for every new thing you want to do.

Dev: Exactly, and the paper focuses on this idea of controlling the execution length instead of just changing the raw actions themselves.

Taro: It sounds like they're giving these complex, long horizon tasks a way to manage their own timing during execution.

Rosa: Right, so they formalize it as learning a separate policy that decides how long each chunk should run, and they train that using off-policy reinforcement learning algorithms.

Dev: That means the adaptation space stays small even though the action chunks get longer, which is a smart way to keep things manageable for real time loops.

Taro: I'm interested in how this affects real world stuff; it shows a consistent preference for shorter executions when the robot is near its target, but longer ones when it’s just moving freely.

Rosa: That’s interesting because it suggests a learned strategy that balances local adjustments with sustained motion across different tasks and even different starting points.

Dev: But what about the safety side? The paper mentions they keep the base policy frozen for now, which means we still have to trust the actions it proposes if something goes really wrong in the real world.

Taro: That's a fair point; they also noted that because they haven't added online behavior cloning yet, the performance is still limited by what those initial frozen actions can actually do.

Rosa: True, so while this shows great potential for sample efficient adaptation in simulation and even some real world tests with just ten minutes of physical interaction, it’s not a plug and play solution yet.

Dev: It points toward the next steps, like incorporating online behavior cloning to help the system recover when the base policy makes a mistake during live interaction.

Rosa: And that leads us right into how this adaptive replanning interacts with things like latency sensitive deployments, which is where we need to keep an eye on next.

Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao

UESTC University · Tongji University · The Hong Kong Polytechnic University

cs.RO

Submitted: 2026-11-10

Updated: 2026-11-10

Project page: https://exrl-project.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies The key insight is to shift adaptation from optimizing actions to…

Key concepts

Execution at the Right Length (ExRL)
This is the core idea where an adapter predicts the optimal length for executing each predicted action chunk. It transforms policy adaptation from optimizing what to do (the action) to deciding how long to do it, keeping the adaptation space low-dimensional.
Execution-Length Policy ($\pi_{ex}$)
This is the learned component that takes the current state and the base policy's output and predicts a feasible execution length. It operates in a discrete space of possible lengths (K), allowing raw actions from the base policy to be executed for variable durations.
Duration-Aware TD Target ($y_t$)
The target signal used to train the critic is modified to account for the chosen execution length ($k_t$). This structure mirrors semiMarkov decision processes, ensuring the critic learns by comparing its prediction against rewards accumulated over variable time steps.
Sample-Efficient Adaptation
ExRL enables adaptation using minimal task-specific tuning and few samples. By optimizing the execution length rather than raw actions, it allows diverse frozen policies to adapt effectively in both simulation and real-world scenarios, often requiring very little physical interaction.

Terminology

Summary

Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies The key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning. p. 0

The Core Concept

ExRL introduces Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. This method shifts adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. p. 0 The adaptation space remains low-dimensional while its expressivity grows with the chunk size. p. 1

Formalization of Adaptation

The method formalizes this idea as reinforcement learning over execution-length decisions. Given a pretrained action-chunked policy πbase with chunk size H, they learn an additional execution-length policy πex: S × AH → ∆(K), where K = K = the set of feasible execution lengths. p. 2 This allows the raw primitive actions to still be produced by πbase, while the learned execution-length policy acts in the much smaller discrete space K. p. 2

Training Signal Construction

The duration-aware TD target is defined as yt = kXt−1i=0γi r(st+i, at+i) + γktV (s′, a′), where kt is the chosen execution length. This target mirrors the Bellman backup used in semiMarkov decision processes (SMDPs). p. 2 The critic learns by minimizing the squared TD error Lq(ϕ) = Eh(Qϕ(st, at:t+H, kt) − yt)2. p. 2

Performance and Learning Patterns

ExRL enables sample-efficient adaptation of diverse frozen policies across simulation and real-world tasks. In real-world experiments, ExRL-Q approaches 100% success on stack cube with only ten minutes of physical interaction. p. 6 The learned execution length patterns show a consistent preference for shorter executions near the object or target than during free motion across tasks and seeds. p. 6

Policy Evolution and Benefits

ExRL can also leverage offline data by treating each H-step trajectory segment as a virtual action chunk, whose prefixes provide replay experience for different execution lengths. Successful online rollouts can be distilled back into the base policy using behavior cloning, improving its chunk predictions. p. 4 The adaptive strategy balances local adjustments and sustained motion to achieve optimal results across different manipulation tasks. p. 1

Limitations and Extensions

The real-world experiments keep the base policy frozen and do not yet incorporate online BC, meaning performance remains constrained by the actions the base policy can propose. Further extensions could involve incorporating online BC to address remaining failures in real-world tasks. p. 6 The work also notes that understanding how adaptive replanning interacts with overlapping inference and execution is an important direction for extending ExRL to latency-sensitive deployments. p. 7

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Sample efficient adaptation of long-chunk policies: ExRL enables sample-efficient adaptation of diverse frozen policies across simulation and real-world tasks with minimal task-specific hyperparameter tuning.

  2. Policy agnostic interface for adaptation: ExRL adds a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms, allowing it to be applied to virtually any class of action-chunked policies without modifying the base policy itself.

  3. Duration aware Q learning: The system learns a critic Qphi(st, at:t+H, k) that estimates the return obtained by executing the first k actions of a predicted chunk and then continuing with the current execution-length strategy, leading to a duration aware TD target.

  4. Leveraging offline data for adaptation: ExRL can leverage general offline trajectories by reinterpreting arbitrary trajectory segments as off-policy experience under different execution lengths, further improving online sample efficiency.

  5. Behavior shaping through learned patterns: ExRL learns that across Lift, Can, Square, and real-world insertion, ExRL selects shorter executions near the object or target than during free motion, concentrating replanning near interaction to balance local adjustments and sustained motion.

  6. Policy refinement via demonstration augmentation: Combining ExRL-Q with Online BC allows for rapid adaptation on tasks like ACT Can by distilling successful online rollouts back into the base policy, reaching near-perfect success competitive with the strongest demonstration based baselines.

  7. Enhanced precision in manipulation: ExRL enables real-world adaptation where it approaches 100% success on stack cube with only ten minutes of physical interaction, and nearly 90% on long horizon clothing folding within 85 minutes, significantly outperforming fixed length execution across all tested tasks.

Sources

Related papers