AI Alignment From Social Choice Perspectives

The gist: AI alignment from human feedback can be analyzed as a formal aggregation pipeline, where social choice theory provides the precise language to express desired properties and identify failure modes in methods like RLHF.

Alignment from Human Feedback and Social Choice Theory

AI alignment is defined as ensuring that artificial intelligence systems act in ways consistent with human intentions, preferences, and normative constraints [1–4]. For open-ended language models, operationalizing these goals presents a difficult challenge because the full range of desirable behaviors cannot be explicitly specified [8]; the underlying human judgments rely on tacit knowledge and subtle tradeoffs between objectives, making them virtually impossible to capture in formal terms [1, 9]. Methods for alignment from human feedback, most notably reinforcement learning from human feedback (RLHF) [12–16], involve annotators evaluating concrete model outputs, and a scoring function trained on this feedback is then used to align the model [2]. This approach implicitly assumes a shared, underlying human intuition that can be statistically recovered by querying human evaluators [21–23]. When annotators inherently disagree, reward modeling goes beyond statistical estimation; it becomes a form of preference aggregation [32]. Evaluating this choice requires examining the assumed model structure and the aggregation method itself, which falls squarely within the domain of social choice theory [33–42].

The Social Choice Perspective on Alignment Pipelines

Social choice provides a formal language to express desired properties explicitly, making the normative assumptions embedded in aggregation procedures more transparent and open to mathematical comparison [33–42]. The paper surveys three related contributions: first, this framing identifies the aggregation rules embedded in widely used methods such as RLHF and DPO [16], showing which assumptions and normative priorities these methods build into their objectives. Second, social choice axioms expose structural failure modes of such rules, identifying conditions under which alignment methods provably violate desirable aggregation properties. Third, social choice provides a broad toolbox of well-studied aggregation methods, informing new approaches that encode different normative objectives through various choices of feedback elicitation, aggregation, and policy optimization [3].

Reward Learning as Implicit Aggregation

When human feedback is heterogeneous, different annotators can give conflicting preference labels for the same prompt. Fitting a single reward then requires the objective to reconcile these judgments into one score per response [3]. BT reward learning is usually read as statistical estimation of latent utilities from noisy pairwise comparisons; however, this framing obscures the reconciliation process, treating it as a technical detail rather than a choice of aggregation rule with inherent normative tradeoffs [3]. The Borda rule, which is closely connected to the underlying aggregation rule of BT reward learning in an idealized setting, favors responses that maximize their expected pairwise advantage against a randomly drawn candidate [3]. Theorem 1 shows that under idealized conditions, the optimizer ranks responses exactly by their Borda scores [61]. This implies that the post-training density ratio against the base policy is, up to a monotone transformation, the πbase-weighted Borda score of y [70].

Clone Robustness and Unanimity Failures

A particularly relevant pathology for language model reward learning is sensitivity to near-duplicate candidates [4]. A clone-sensitive BT model can treat the number of variants in a cluster as meaningful, so adding near-duplicates can change the rewards assigned to individual candidates and the total policy probability assigned to the cluster as a whole [84]. The corresponding formal requirement is robustness to approximate clones: the learned reward should be stable under the addition of near-duplicates [85]. Theorem 3 shows that standard BT reward learning is not robust to approximate clones, but a weighted version of the population BT objective can be robust if it minimizes Lwλ(r) [85]. Furthermore, when the reward is required to generalize through a restricted reward class RΘ, unanimity—the property where every annotator strictly prefers response y+ to y− implies rθ(y+) > rθ(y−)—can fail [38]. Theorem 5 shows that under certain conditions on the loss function, there exist responses where every annotator prefers y+ to y−, but the learned reward ranks them in the opposite order: rθ⋆ (y−) > rθ⋆ (y+) [38].

Direct Alignment and Policy Optimization

The paper explores how alignment objectives can keep the pairwise structure directly when moving beyond a scalar reward. Maximal lotteries address intransitivity by comparing candidate distributions using the population majority margin, which corresponds to the maximin strategies of a zero-sum game [6]. Nash Learning from Human Feedback (NLHF) applies this maximin idea directly at the level of policies, choosing a policy with the best worst-case comparison probability against alternative policies [107]. Theorem 7 demonstrates that when NLHF is used, the maximal lottery attains the smallest possible worst-case distortion among rules observing anonymous BT pairwise feedback [90]. This result extends to KL-regularized objectives where the relevant comparison is with policies that are allowed to move the same distance from the reference policy [90].

Welfare Loss and Richer Elicitation

The welfare loss perspective examines what is lost due to limited information, showing that identical pairwise observation data can arise from populations with fundamentally different preferences in terms of welfare [88]. The utilitarian distortion measures the worst-case welfare loss that a policy can incur over normalized reward profiles consistent with the elicited feedback [90]. Theorem 6 bounds this distortion for BT responses under a specific temperature parameter η: (1 − o(1)) η ≤ sup Px Distx (π0(Px); Px) ≤ O(η 2) [90]. Furthermore, richer elicitation, such as eliciting two comparisons per annotator, can yield enough information to completely identify the voter type distribution and optimize for any desired social welfare function [94].

Discussion on Future Directions

The discussion points to several related research threads extending this pipeline view. Preference collapse occurs when fine-tuning a base policy toward a learned reward signal can narrow the policy’s behavior, amplifying biases in the preference data [115, 120]. Pluralistic alignment involves modeling multiple perspectives in parallel, such as training multiple reward models matched to distinct segments of the population [124]. Personalization allows aligning objectives to individual annotators by placing each user’s reward in a shared low-dimensional space [139]. Ultimately, the work uses social choice theory to diagnose how alignment pipelines aggregate heterogeneous feedback and design learned rewards or policies that handle disagreement more explicitly.

--- Page 18 ---

[1] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable Agent Alignment via Reward Modeling: A Research Direction, 2018. arXiv:1811.07871 [2].

[2] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Jiayi Zhou, Borong Zhang, Donghai Hong, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai [2].

[3] Iason Gabriel. Artificial Intelligence and Values and Alignment [3].

[4] Stuart J. Russell. Human Compatible: Artificial Intelligence and the Problem of Control [4].

[5] Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for Advanced Machine Learning Systems [5].

[6] Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of Language Agents [6].

[7] Iason Gabriel and Geoff Keeling. A Matter of Principle? AI Alignment as the Fair Treatment of Claims [7].

[8] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved Problems in ML Safety [8].

[9] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J´er´emy Scheurer [9].

[10] Andrew P. Clark, Kate L. Howard, Andy T. Woods, Ian S. Penton-Voak, and Christof Neumann. Why Rate When You Could Compare? Using the “EloChoice” Package to Assess Pairwise Comparisons of Perceived Physical Strength [10].

[11] Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-Armed Dueling Bandits Problem [11].

[12] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences [12].

[13] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright [13].

[14] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford [14].

[15] Nisan Stiennon et al. Learning to Summarize with Human Feedback [15].

[16] Rafael Rafailov et al. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model [16].

[17] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion [17].

[18] Saffron Huang et al. Collective Constitutional AI: Aligning a Language Model with Public Input [18].

[19] Maarten Buyl et al. AI Alignment at Your Discretion [19].

[20] Abigail Z. Jacobs and Hanna Wallach. Measurement and Fairness [20].

1. textbf Citation of Aggregation Rules in Reward Learning (Section 3): The model can be designed to explicitly encode Borda-like aggregation rules by minimizing a loss function like Equation (4), which represents the asymptotic limit of empirical BT learning given infinite comparisons and a fully expressive reward class. This allows for explicitly principled ways to handle disagreement rather than implicit aggregation.

2. textbf Robustness to Approximate Clones (Section 4): The learned reward can be made stable under near-duplicate responses by minimizing the weighted BT loss, which is robust because it assigns weights based on the normalized mass of points in the response space for which y is the nearest candidate, satisfying Definition 2 of robustness.

3. textbf Unanimity Guarantee Preservation (Section 5): The system can be constrained to satisfy unanimity by choosing a loss function with specific properties, as Theorem 5 shows that for certain loss functions, "every annotator prefers y+ to y−, but the learned reward rθ ranks them in the opposite order: rθ⋆ (y−) > rθ⋆ (y+)" is prevented.

4. textbf Maximal Lottery Optimization (Section 7): The policy can be optimized using Nash Learning from Human Feedback (NLHF), which uses a maximin idea directly at the level of policies, to achieve an optimal worst-case distortion bound of 1/2 + o(1)η, ensuring the policy maximizes welfare against alternative policies.

5. textbf Pluralistic Alignment via Reward Ensembles (Section 8): The system can model preference heterogeneity by training a compact reward ensemble whose aggregate pairwise choices match the observed preference Px, so each component induces a coherent policy while the mixture preserves disagreement, allowing for Overton-style slates or mixture sampling preserves population-level diversity.

6. textbf Personalized Alignment via Low-Rank Reward Modeling (Section 8): Individual annotator preferences can be incorporated by placing each user’s reward in a shared low-dimensional space, enabling lightweight user embeddings or low-rank adapters then specialize a shared model to the individual.

Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.