Toward a Theory of Value in AI Alignment

The paper Toward a Theory of Value in AI Alignment investigates the implicit theories of human values that underpin the field of AI value alignment. The authors conducted a qualitative analysis of 94 AI alignment research papers, sampled via a snowball method from four canonical seed papers, to discern how the field conceptualizes, operationalizes, and evaluates human values.

The paper's central findings are that the majority of papers in the dataset do not define what they mean by values, with 79% neither defining nor describing the term. Instead, the field heavily relies on preferences as a stand-in for values, with 82% of papers treating preferences as a substitute for values. This approach, which the authors term preferentist, is rooted in economic theory, specifically rational choice theory and expected utility maximization. The authors found that 87% of papers implicitly or explicitly use utility maximization as part of their technical framework, and 90% treat values as measurable.

The paper critiques this dominant paradigm for several reasons. It argues that the reliance on preferences reduces complex, culturally situated concepts to binary choices, and that the field abstracts away questions of whose values are represented, with 91% of papers not referring to specific groups of humans whose values should be followed. The authors also note a trend toward alignment without humans, where researchers use synthetic data and LLM-as-a-judge approaches instead of human annotators, potentially closing off alternative methods for contesting and enacting values. Additionally, 53% of papers treated values as static, and 66% presented thin descriptions of values that are instrumental and calculative rather than situated in societal context.

The paper's contributions include developing a framework for analyzing the theory of value in AI alignment, conducting a qualitative study of the literature, forming an initial hypothesis about where the field's theory of value sits, and outlining alternative theories of human values from anthropology and psychology as potential paths forward. The authors conclude that AI alignment lacks a concrete theory of human values and implicitly adopts beliefs rooted in economic theory, which limits the possible range of human values with which AI can be aligned. They call for greater interdisciplinary engagement to bring specificity and under-explored perspectives into the debate on whether and how AI can address human values.

Improvements to AI Systems:

1. Value-Definition Engine: The AI system will explicitly define and operationalize values before any alignment task, distinguishing between preferences, norms, and moral principles. It will refuse to proceed if values are undefined, and will generate a formal specification of whose values (e.g., demographic, cultural, or stakeholder group) are being targeted, avoiding the current 91% failure to reference specific human groups.

2. Preference-Contextualizer: The system will no longer treat preferences as binary, utility-maximizing choices. Instead, it will map each preference to its social, cultural, and situational context, using a multi-dimensional value model (e.g., from anthropology) that includes relational, symbolic, and non-instrumental dimensions. This prevents reduction of complex values to simple rankings.

3. Human-in-the-Loop Value Contestation Module: The system will incorporate mandatory human annotators from diverse backgrounds for value-sensitive decisions, and will actively flag cases where synthetic data or LLM-as-a-judge is used, requiring justification. It will also provide a mechanism for end-users to contest and re-negotiate the values encoded in the system, countering the alignment without humans trend.

4. Dynamic Value Updating: The system will treat values as non-static, updating its value model based on new human feedback, societal shifts, or contextual changes. It will track value drift over time and alert developers when the original alignment target becomes outdated, addressing the 53% static-value problem.

5. Thick Description Generator: For any alignment objective, the system will generate a thick description of the values involved—including historical, cultural, and ethical narratives—rather than thin, calculative metrics. This will be used to audit whether the system's optimization criteria truly capture the richness of human values, and to propose alternative value frameworks (e.g., from psychology or anthropology) when the current one is too narrow.

6. Interdisciplinary Value Advisor: The system will automatically consult and integrate theories from anthropology, psychology, and philosophy (e.g., value pluralism, capability approach) when designing alignment objectives. It will flag when its underlying assumptions match rational choice theory and suggest alternative models, ensuring the system is not locked into a single economic paradigm.

What the improved AI system can do: It can align with human values in a way that is explicitly defined, culturally situated, contestable, dynamic, and interdisciplinary—avoiding the reductionist, preference-based, utility-maximizing default that dominates current practice. It will produce alignment outcomes that are more robust, ethically defensible, and responsive to diverse human populations.

Can AI systems be aligned to human values? The popularization of large language models (LLMs) and multi-modal foundation models has seen a rise in harms spanning from toxic speech and hallucinations to AI agents executing unauthorized actions. Within the field of AI safety, these harmful instances are often framed as the alignment problem, or of models being misaligned with human values. Researchers have responded by pursuing applied and theoretical AI value alignment efforts, often without specifying what they mean by human values. How does the field of AI value alignment conceive of human values? How are these conceptions of values technically operationalized and evaluated? What does the emergent theory of value from this field signify for the future of AI? We annotated 94 value alignment research papers to discern their implicit theory of values in AI. The majority do not define values, relying heavily on preferences as a stand in that runs the risk of reducing complex culturally situated concepts down to binary choices. As researchers dispense with using human annotators for model training and evaluation, turning instead to synthetic data and autorater approaches to aligning and evaluating models, we identify the potential to close off alternative methods for contesting and enacting values in foundation models. In making AI value alignments philosophical commitments explicit, we seek to bring great specificity and under explored perspectives in the debate on whether and how AI can address human values.