Recurrent Reinforcement Learning with Memoroids
cs.LG, cs.AI
Submitted: 2024-02-15
Updated: 2026-08-27
Comments: Accepted to NeurIPS 2024
Code: https://github.com/oogle/jax
License: http://creativecommons.org/licenses/by/4.0/
The gist: Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states.
Terminology
Abstract
Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states. Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models called Linear Recurrent Models. We discover that the recurrent update of these models resembles a monoid, leading us to reformulate existing models using a novel monoid-based framework that we call memoroids. We revisit the traditional approach to batching in recurrent reinforcement learning, highlighting theoretical and empirical deficiencies. We leverage memoroids to propose a batching method that improves sample efficiency, increases the return, and simplifies the implementation of recurrent loss functions in reinforcement learning.
Sources
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Mastering Diverse Domains through World Models
- CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms
- Exploring the Promise and Limits of Real-Time Recurrent Learning
- RWKV: Reinventing RNNs for the Transformer Era
- Prioritized Experience Replay
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks