Reward-Conditioned Reinforcement Learning
cs.LG
Submitted: 2026-03-05
Updated: 2026-11-10
Comments: NeurIPS 2026
Code: https://github.com/DAVIAN-Robotics/SimbaV2
License: http://creativecommons.org/licenses/by/4.0/
The gist: Sample-efficient reinforcement learning typically uses a fixed reward function both to guide exploration and to supervise learning.
Terminology
Abstract
Sample-efficient reinforcement learning typically uses a fixed reward function both to guide exploration and to supervise learning. However, the same experience can provide supervision under multiple related reward specifications, even when the objective of interest remains fixed. We introduce Reward-Conditioned Reinforcement Learning (RCRL), an off-policy approach that collects experience under a nominal objective while training a reward-conditioned agent using counterfactual rewards recomputed from replay. RCRL constructs alternative reward specifications by perturbing the nominal reward, providing diverse supervision without additional environment interaction. Across single-task, multi-task, and vision-based control benchmarks, RCRL improves nominal task performance and sample efficiency when integrated with strong RL backbones. As a secondary capability, the learned policies support zero-shot behavioral adjustment toward related objectives without further training. These results demonstrate that diverse reward supervision can improve learning of a predefined objective while also enabling policy steerability.
Sources
- Solving Rubik's Cube with a Robot Hand
- Is Temporal Difference Learning the Gold Standard for Stitching in RL?
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Reward-Conditioned Policies
- Continuous control with deep reinforcement learning
- nGPT: Normalized Transformer with Representation Learning on the Hypersphere
- Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
- XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
- Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
- OGBench: Benchmarking Offline Goal-Conditioned RL
- Horizon Reduction Makes RL Scalable
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control
- HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation
- Fast Adaptation with Behavioral Foundation Models
- DeepMind Control Suite
- Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models
- Does Zero-Shot Reinforcement Learning Exist?
- MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning