Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models
cs.LG, cs.CL
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: Accepted to NeurIPS 2026
Code: https://github.com/Martin-qyma/Rethinking-Personalized-Generation
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users.
Terminology
Abstract
Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empirical studies are first used to reveal the existence of a massive, untapped performance headroom for personalized generation through test-time alignment. We demonstrate that personalized generation is uniquely suited for test-time scaling methods like Best-of-N (BoN) because it can be viewed primarily as a candidate matching problem rather than a generator capability bottleneck. While reward models could in principle exploit this headroom, they are poorly calibrated for personalization, and their billion-parameter scale makes scoring large candidate pools prohibitively expensive. To overcome this limitation, we propose a parameter-efficient framework utilizing million-parameter scale multi-layer perceptron (MLP) ranking models. Our personalized ranking model directly reuses the internal embeddings of the base generator with minimal overhead. By scaling train-time data to provide fine-grained personalized preferences, this million-parameter ranking model accurately scores large candidate pools and can seamlessly guide generation to reduce the cost of materializing N candidates. Extensive experiments on nine datasets spanning three personalized generation settings show that our personalized ranking model effectively exploits the discovered headroom, outperforming billion-parameter generalist reward models on every dataset, with under 0.4% of their parameters and four orders of magnitude lower scoring latency.
Sources
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- LoRe: Personalizing LLMs via Low-Rank Reward Modeling
- PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
- The Era of Real-World Human Interaction: RL from User Conversations
- ARGS: Alignment as Reward-Guided Search
- Cascade Reward Sampling for Efficient Decoding-Time Alignment
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
- Decoupled Weight Decay Regularization
- Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
- Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
- T-POP: Test-Time Personalization with Online Preference Feedback
- Qwen2.5 Technical Report
- Language Model Personalization via Reward Factorization
- A Roadmap to Pluralistic Alignment
- Instant Personalized Large Language Model Adaptation via Hypernetwork
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- BERTScore: Evaluating Text Generation with BERT
- Group Preference Optimization: Few-Shot Alignment of Large Language Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks