ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
cs.LG, cs.RO
Submitted: 2026-05-10
Updated: 2026-09-27
Code: https://github.com/ColinQiyangLi/qc
License: http://creativecommons.org/licenses/by/4.0/
The gist: Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman
Terminology
Abstract
Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes. We prove that the ACSAC Bellman operator is a γ-contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves strong performance on long-horizon, sparse-reward manipulation and navigation tasks across both offline RL and offline-to-online RL settings.
Sources
- TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapse
- IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
- AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization
- CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- SEAR: Sample Efficient Action Chunking Reinforcement Learning
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Chunk-Guided Q-Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks