Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback
cs.LG, cs.AI, stat.ME, stat.ML
Submitted: 2024-03-08
Updated: 2026-09-07
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pluralistic alignment requires learning from feedback that reflects persistent and potentially conflicting stakeholder preferences while ultimately selecting a single collective policy.
Terminology
Abstract
Pluralistic alignment requires learning from feedback that reflects persistent and potentially conflicting stakeholder preferences while ultimately selecting a single collective policy. We study this problem in offline reinforcement learning from human feedback (RLHF), where the party associated with each comparison is observed. Under a shared low-rank linear reward model, we jointly estimate party-specific rewards and perform pessimistic policy optimization under Nash, Utilitarian, and Egalitarian social-welfare objectives. We establish nonasymptotic bounds for party-specific reward estimation and the resulting policy suboptimality under offline coverage conditions. We further consider general pairwise preferences that need not admit a scalar reward representation and may exhibit cycles. In this setting, we construct a pessimistic von Neumann winner policy and derive corresponding performance guarantees. Under these models, our results provide a unified finite-sample solution to a central challenge in pluralistic alignment: learning from limited, heterogeneous, and potentially cyclic feedback, and producing a single policy with explicit collective-welfare guarantees. Our framework thereby makes preference aggregation an explicit and statistically analyzable design choice rather than an implicit consequence of pooling human feedback.
Sources
- When is Offline Two-Player Zero-Sum Markov Game Solvable?
- Few-Shot Learning via Learning the Representation, Provably
- Generative Social Choice
- Multi-dimensional domain generalization with low-rank structures
- Whose Opinions Do Language Models Reflect?
- Is RLHF More Difficult than Standard RL?
- Provable Offline Preference-Based Reinforcement Learning
- Proportional Fairness in Federated Learning
- Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons
- Fine-Tuning Language Models from Human Preferences
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks