Bayesian Preference Learning for Test-Time Steerable Reward Models
The paper introduces Variational In-Context Reward Modeling (ICRM), a new Bayesian method for reward modeling that allows language models to adapt their preferences at test time using in-context examples. ICRM works by treating reward as a latent variable and using in-context demonstrations to infer a preference distribution. This enables steerability for single and multi-objective tasks, improving performance on benchmarks and showing practical use in reinforcement learning.