Unbiased Reward Modeling from Implicit Feedback for LLM Alignment
ImplicitRM proposes a method to train unbiased reward models from ambiguous human preference data by stratifying samples into four latent groups: positive-active, negative-active, positive-passive, and negative-passive. This framework uses likelihood maximization to derive a theoretically unbiased learning objective that overcomes issues like missing negative samples and user bias.