Pointwise or Pairwise: When Do Pairwise Losses Help Reward Learning, Provably?
cs.LG, stat.ML
Submitted: 2026-09-29
Updated: 2026-09-29
Comments: 58 pages, 4 figures, 3 tables; second through sixth authors contributed equally
License: http://creativecommons.org/licenses/by/4.0/
The gist: Pairwise losses are increasingly used for reward learning even when pointwise rewards are observed, with mixed empirical results.
Terminology
Abstract
Pairwise losses are increasingly used for reward learning even when pointwise rewards are observed, with mixed empirical results. When and why do pairwise losses outperform pointwise losses? We study this question in a grouped offline contextual-bandit setting allowing multiple actions per context, capturing many reward learning scenarios. We compare Value Regression (VR), which regresses observed rewards pointwise, with Value Difference Regression (VDR), which regresses reward differences between a pair of actions sampled under the same context. We consider a semiparametric model where the mean reward is the sum of a learnable action-dependent component and an arbitrary context-dependent yet action-independent nuisance, capturing context-specific disturbances. Using a unified localized analysis, we prove finite-sample regression guarantees for finite and linear function classes and translate them into offline-regret bounds. For finite classes, VDR eliminates the misspecification term in the VR bound and improves a reward-scale-dependent error term by averaging over actions within each context, a benefit absent from the corresponding VR term. For linear classes, neither method uniformly dominates: within-context differencing removes nuisance-induced bias but may increase estimation variance relative to using absolute rewards when the misspecification is sufficiently low. This yields a feature geometry-dependent bias-variance tradeoff, which we corroborate with numerical experiments.
Sources
- Program Synthesis with Large Language Models
- Training Verifiers to Solve Math Word Problems
- Qwen2.5-Coder Technical Report
- Reward Learning from Best-of-$N$ Preference Data: Targets, Tradeoffs, and Design Principles
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Comments on the Du-Kakade-Wang-Yang Lower Bounds
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks