Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons

The paper Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons addresses the problem of learning item rewards from pairwise comparisons when the comparisons are elicited from crowdworkers of varying reliability. The authors adopt the Boltzmann-rational model, which extends the Bradley–Terry–Luce (BTL) model by incorporating worker competencies. In this model, each item j has a latent reward rj, and each worker s has a competency parameter βs ∈ [−1, 1]. The probability that worker s prefers item w over item l is given by P [w ≻ l s] = σ(βs (rw − rl)), where σ is the logistic function. This model captures a spectrum of worker behaviors: βs = 1 corresponds to an ideal annotator, βs = 0 to a random spammer, and βs = −1 to an adversarial worker. Given a dataset D = (wi, li, si) N i=1 of N pairwise comparisons, the goal is to jointly estimate the reward vector r ∈ RK and the competency vector β ∈ [−1, 1]M.

The authors derive an expectation-maximization (EM) algorithm for inference under this model. To derive the EM algorithm, they use a Pólya-Gamma augmentation, which introduces auxiliary variables to transform the logistic likelihood into a conditionally Gaussian form. This leads to a quadratic objective for the M-step which enables efficient alternating maximization over β and r in closed form. The E-step computes the conditional expectation of the Pólya-Gamma latent variables given the current parameter estimates, resulting in a simplified Q function. The M-step uses alternating maximization, where worker competencies are updated via closed-form aggregated statistics, and item rewards are obtained by solving a linear system Hr = b with the constraint that rewards sum to zero.

To address the non-convexity of the M-step optimization, the authors show that it can be cast as a rank-1 matrix sensing problem. Specifically, defining X = βr⊤ ∈ RM ×K and a linear operator A: RM ×K → RN, the M-step reduces to minimizing ∥A(βr⊤)−u∥22. They establish that the operator A satisfies the Restricted Isometry Property (RIP) given a sufficient number of samples. Leveraging results from non-convex low-rank matrix recovery, they prove that under the RIP condition, at each iteration of the EM algorithm, every local minimum of the M-step objective is near-global minimum. Using standard results for EM, they show that the likelihood increases monotonically and converges to a stationary point of the likelihood function.

The authors conduct extensive experiments on both real-world and synthetic datasets. On synthetic datasets, they vary the number of measurements N, the number of workers M, and the number of items K individually while keeping the other two constant. They sample β uniformly from [−1, 1] to include adversarial workers. The results show that methods not modeling annotator competence (BT, BARP, RC) perform poorly in the presence of adversarial workers, while methods that explicitly model annotator competence (BoRaEM, HBTL, CrowdBT) achieve superior performance in denser regimes. BoRaEM is shown to be more robust than CrowdBT in sparse measurement settings. When varying the number of workers, BoRaEM remains quite stable while CrowdBT's performance decays more significantly. When varying the number of items, BoRaEM and HBTL degrade slower than CrowdBT under larger N regimes.

For real-world datasets, the authors use the FaceAge dataset (IMDB-WIKI-SbS) and the Reading Difficulty (Passage) dataset. On the FaceAge dataset, BoRaEM achieves the best performance, followed by HBTL and CrowdBT. On the Passage dataset, CrowdBT achieves the highest performance, followed by HBTL and BoRaEM, though all methods achieve relatively low performance due to the small dataset size and the inherent difficulty of modeling preferences in reading difficulty.

The authors also evaluate robustness under adversarial conditions by injecting synthetic spammers of four types: random choosers, position-biased spammers, malicious adversaries, and a combined setting with all spammer types in equal proportion. As the spammer proportion increases from 0% to 44.4%, BoRaEM, HBTL, and HTCV consistently perform the best, followed by CrowdBT and FactorBT, while BARP, BT, and RC exhibit substantial performance drops. These results highlight the importance of explicitly modeling annotator competencies in crowdsourcing.

The paper concludes that the proposed BoRaEM algorithm provides a unified framework capable of handling diverse spammer behaviors—from random clickers to malicious adversaries—offering a practical and theoretically grounded alternative for real-world crowdsourcing platforms where worker quality is heterogeneous and unknown a priori. Future work includes designing models for robust RLHF that can jointly predict item feature importance vectors while effectively identifying and mitigating the influence of spammers.

Based on this paper, I can improve AI systems in the following specific ways:

1. Robust Reward Learning for RLHF (Reinforcement Learning from Human Feedback)

• Improvement: Replace standard Bradley-Terry-Luce models with the competency-aware Boltzmann-rational model that jointly estimates item rewards and per-annotator reliability (β ∈ [-1, 1]).

• What the improved system can do: Automatically detect and down-weight adversarial or random human feedback during reward model training, preventing malicious or careless annotators from corrupting the learned reward function. It can maintain high reward accuracy even when up to 44% of annotators are spammers, which current RLHF pipelines fail to handle.

2. Adaptive Crowdsourcing Quality Control

• Improvement: Integrate the EM algorithm with Pólya-Gamma augmentation to provide real-time, closed-form updates of worker competency scores without requiring separate validation tasks or gold-standard questions.

• What the improved system can do: Dynamically re-weight contributions from each crowdworker as more comparisons are collected, automatically identifying and neutralizing position-biased spammers (who always click the left option) and malicious adversaries (who deliberately invert preferences). The system can flag low-competency workers in real-time and adjust their influence on the final ranking.

3. Sparse-Data Preference Learning

• Improvement: Use the rank-1 matrix sensing formulation with RIP guarantees to enable reliable reward estimation even when the number of comparisons is far smaller than the number of items × workers.

• What the improved system can do: Learn accurate item rankings from very sparse pairwise comparison datasets (e.g., only a few comparisons per item) where existing methods like CrowdBT degrade significantly. This is critical for new products, niche domains, or early-stage data collection where obtaining dense annotations is expensive.

4. Unified Spammer-Proof Ranking System

• Improvement: Implement the alternating M-step that solves a linear system Hr = b for item rewards and closed-form updates for worker competencies, with the sum-to-zero constraint ensuring identifiability.

• What the improved system can do: Provide a single, theoretically grounded pipeline that handles mixed annotator populations (ideal, random, adversarial) without requiring prior knowledge of worker types. It can automatically calibrate the scale of rewards across different worker groups, preventing adversarial workers from skewing the entire ranking.

5. Theoretical Convergence Guarantees for Non-Convex Optimization

• Improvement: Leverage the RIP-based proof that every local minimum of the M-step is near-global, combined with monotonic likelihood increase, to ensure stable training.

• What the improved system can do: Guarantee that the reward learning algorithm will not get stuck in poor local optima, even with non-convex objectives. This makes the system reliable for production deployment where training stability and reproducibility are essential, and it can be safely used in iterative human-in-the-loop feedback cycles.

6. Transferable Worker Reliability Profiles

• Improvement: Store and reuse learned competency vectors (β) across different tasks or datasets from the same worker pool.

• What the improved system can do: Build a persistent profile of each annotator's reliability across multiple tasks. When a new task begins, the system can initialize worker competencies from historical data, requiring fewer comparisons to achieve accurate rankings and immediately suppressing known spammers before they cause damage.

The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley-Terry-Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified Q function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings. The code and data is publicly available at https://github.com/KaustubhShejole/BoRa EM.