PoEM: Predicting RL Outcomes from Existing Policies
cs.LG, cs.AI, cs.CL, cs.CV
Submitted: 2026-09-24
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Foundation models are post-trained with reinforcement learning (RL) to maximize specific rewards, such as human alignment, correctness, or instruction following.
Terminology
Abstract
Foundation models are post-trained with reinforcement learning (RL) to maximize specific rewards, such as human alignment, correctness, or instruction following. This post-training process is computationally intensive, sometimes unstable, and has to be run from scratch every time the reward model changes or when we want to combine multiple rewards. We hence ask: given a new reward function, is it possible to predict the RL outcomes without actually running RL on it? We answer this in the affirmative by introducing PoEM, a framework to predict the outputs of RL on a new reward function using a set of models already post-trained on other rewards. First, we show that if the new reward function can be written as a linear combination of existing ones, then the new policy in log-space can be written as a linear combination of the existing log-policies. Surprisingly, even in cases where the rewards are not linearly connected, we observe that often log-policies from RL training span an approximately low-rank subspace across rewards. To our benefit, the weighting coefficients for this combination can be estimated using only the reward or basis policy outputs on the samples. We turn these observations into an algorithm that takes post-trained models and a new reward function, and approximates the target RL policy without actually running any additional RL training. We experimentally validate our approach across synthetic and real rewards, spanning both text and image modalities.
Sources
- Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
- Successor Features for Transfer in Reinforcement Learning
- Training Diffusion Models with Reinforcement Learning
- InternLM2 Technical Report
- Bootstrapping Language Models with DPO Implicit Rewards
- Training Verifiers to Solve Math Word Problems
- Process Reinforcement through Implicit Rewards
- RLHF Workflow: From Reward Modeling to Online RLHF
- Editing Models with Task Arithmetic
- Datamodels: Predicting Predictions from Training Data
- Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
- PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training
- Tuning Language Models by Proxy
- Decoding-time Realignment of Language Models
- AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
- WebGPT: Browser-assisted question-answering with human feedback
- Training language models to follow instructions with human feedback
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks