Procedural Fairness Failures in RLHF from Preference Averaging

Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneous, this aggregation induces a procedural fairness failure where majority preference groups dominate reward learning while minority preferences are systematically under-represented. This work defines procedural fairness in alignment as preserving distinct preference signals during reward modeling and shows that standard RLHF violates this via preference averaging. Preference-Aware RLHF (PA-RLHF) is introduced, separating optimization across preference modes at the reward learning stage. In a controlled setting, PA-RLHF improves overall alignment accuracy from 46.9% to 67.9% and reduces the fairness gap between best and worst aligned groups from 15.9 to 9.6 percentage points. These results show that procedural fairness failures in alignment can arise from structural design choices in reward learning, even in controlled, noise-free settings, with direct implications for large language models and agentic systems, where biased reward models can compound inequities across sequential decisions.

Standard RLHF optimizes a single reward model over aggregated preference data, causing preference influence to scale with dataset frequency. Alignment behavior thus reflects prevalence alongside content, enabling majority preferences to dominate while systematically under-optimizing minority groups. This constitutes a procedural fairness failure, as the alignment objective itself disadvantages minority preferences even when groups are internally consistent. In agentic systems, where reward models guide multi-step planning and action, such early aggregation can repeatedly bias subsequent decisions, leading minority preferences to be persistently ignored over time.

PA-RLHF intervenes at the reward learning stage of RLHF by preventing heterogeneous preferences from collapsing into a single optimization objective. The alignment process preserves distinct preference signals during reward learning. The study evaluates PA-RLHF in a controlled setting isolating preference aggregation effects under heterogeneity. The dataset contains 971 pairwise comparisons from 60 simulated raters across 20 prompts. Raters are programmatically assigned to three groups of 20, each with a distinct preference profile: concise responses (20-35 words, 85% within-group consistency), detailed responses (60-90 words, 85% consistency), and technical/formal responses (40-60 words, 80% consistency). Each rater evaluated 15-18 randomly sampled prompts. This controlled simulation isolates preference aggregation effects from annotation noise and demographic confounds.

Repeated preference comparisons reveal three latent preference groups characterized by internally consistent yet mutually conflicting alignment criteria—for example, preferences favoring concise answers over detailed explanations, or technical rigor over stylistic flexibility. Each feedback instance consists of a prompt, two candidate responses, and a binary preference label. Data are split by raters using a 75/25 train-test partition. Ground-truth group assignments are used exclusively for evaluation. The base language model is held fixed throughout reward learning to ensure that observed effects arise from preference aggregation over representation learning.

PA-RLHF modifies reward learning by explicitly separating optimization across inferred preference modes. Each preference comparison is embedded using fixed sentence-level representations over prompt–response pairs. Representations are clustered into k = 3 preference modes, selected via silhouette score sweep over k = 2–6 on held-out data (silhouette = 0.199, ARI = 0.443). The low silhouette score reflects that preference groups differ stylistically, not semantically, producing soft geometric boundaries in embedding space. ARI = 0.443 confirms meaningful alignment with ground-truth group structure, supporting k = 3 as a principled choice. Fixing the number of modes isolates procedural effects from errors in preference discovery. A separate reward model is trained for each mode using only the feedback assigned to that cluster, with a simple parametric model to isolate procedural effects and avoid confounding model capacity. Reward learning proceeds independently within each mode, ensuring that optimization updates for one preference group are not influenced by the relative frequency of other, potentially conflicting preferences. Clustering is used only to separate preference signals; the observed improvements arise from eliminating cross-group interference during reward learning, not from clustering itself.

Prompt-response pairs are embedded using all-MiniLM-L6-v2 (SBERT, 384-dim, frozen weights). User feature vectors are the mean of preferred-response embeddings concatenated with a response-length scalar (385-dim), normalized to zero mean and unit variance. Clustering uses K-Means with 20 restarts (seed 42). The reward model is Logistic Regression trained on the concatenation of the prompt embedding and the difference between response embeddings (768-dim total), approximating Bradley-Terry pairwise preference. Alignment is evaluated by scoring candidate responses using the corresponding preference-specific reward model, without performing full policy optimization. This separation ensures that minority preference modes receive comparable optimization attention to majority modes during reward learning, preventing dominance through aggregation. By separating reward optimization across preference modes, PA-RLHF ensures that no group’s influence is diminished solely due to lower prevalence, enforcing procedural fairness directly at alignment time.

Procedural fairness is assessed through group-level alignment accuracy disaggregated by preference group. Alignment accuracy measures agreement with group-specific preference judgments. Standard RLHF exhibits substantial group-level imbalance despite moderate aggregate performance. While overall alignment accuracy reaches 46.9%, the most aligned preference group achieves 56.2% accuracy, whereas the least aligned groups reach only 40.3-41.2%, producing a fairness gap of 15.9 percentage points between best and worst-aligned preference groups. PA-RLHF directly addresses this outcome by separating reward learning across preference modes, reducing the fairness gap to 9.6 percentage points (a 40% reduction). While PA-RLHF substantially reduces the fairness gap, the residual 9.6 pp difference suggests that clustering-based separation alone does not eliminate all group-level misalignment, motivating further procedural refinements. Minority preference groups experience large absolute gains in alignment accuracy (+27.6 and +32.8 percentage points), while the majority group improves only marginally (+7.3 percentage points). These results show that aggregation reallocates optimization pressure toward majority preferences as a consequence of objective construction, producing predictable minority under-alignment even in noise-free settings.

Procedural fairness in alignment requires that no preference group is systematically disadvantaged by the construction of the learning objective itself. Standard RLHF enforces a single normative objective over inherently pluralistic preferences, producing a procedural fairness failure that systematically under-aligns minority preference groups. Downstream interventions address symptoms after deployment, whereas the inequities observed here originate upstream, at design time, through the construction of the learning objective itself. Preserving preference heterogeneity during reward learning offers a principled alternative, suggesting that equitable alignment requires explicit structural accommodation of diversity rather than reliance on averaged objectives that obscure minority signals. The goal of this work is diagnostic and scope-limited: to isolate a procedural failure mode and establish its role in group-level misalignment within RLHF. The controlled setting isolates procedural effects but limits generalizability beyond this diagnostic context. This study uses controlled preference data, fixed clustering assumptions, and intervenes only at the reward learning stage. These choices clarify aggregation’s role in group-level misalignment but defer questions of mode interpretability, adaptive clustering, and end-to-end deployment dynamics to future work. Future work should extend these findings to production preference datasets and motivate procedural audits of alignment pipelines prior to deployment, particularly for agentic systems whose behavior compounds over time.

Improvements to AI Systems:

1. Preference-Mode-Aware Reward Modeling: Replace single reward models with multi-modal reward architectures that cluster preference feedback into distinct modes (e.g., via embedding-based clustering) and train separate reward heads per mode. This prevents majority preference dominance during RLHF.

2. Procedural Fairness Auditing at Training Time: Integrate group-disaggregated alignment accuracy metrics (e.g., fairness gap between best/worst-aligned preference groups) into the reward learning pipeline as a training-time diagnostic, enabling early detection of structural bias before deployment.

3. Frequency-Decoupled Optimization: Modify the reward learning objective to normalize optimization pressure across preference clusters (e.g., by weighting loss contributions per cluster inversely to cluster size), ensuring minority preferences receive comparable learning signal regardless of dataset prevalence.

4. Preference-Signal Preservation Layer: Add a preprocessing step that embeds prompt–response pairs, clusters them into preference modes, and routes each pairwise comparison to mode-specific reward models—preventing cross-group interference during gradient updates.

5. Deployment-Time Preference Routing for Agentic Systems: For multi-step decision-making, use the learned preference-mode classifiers to dynamically route reward scoring per user or per context, ensuring that sequential decisions do not compound majority bias over time.

What the Improved AI System Can Do:

• Achieve higher overall alignment accuracy (e.g., 67.9% vs. 46.9% in controlled settings) by preserving distinct preference signals rather than averaging them.

• Reduce the fairness gap between best- and worst-aligned user groups by 40% (e.g., from 15.9 to 9.6 percentage points), with minority groups gaining +27–33 percentage points in alignment accuracy.

• Maintain consistent alignment quality across heterogeneous user populations, even in noise-free settings where standard RLHF systematically under-optimizes minority preferences.

• In agentic systems, avoid persistent neglect of minority preferences across sequential decisions by ensuring each step’s reward reflects the user’s specific preference mode, not a majority-aggregated norm.

• Provide a procedural audit trail: before deployment, system designers can inspect per-mode reward model performance and fairness gaps, enabling preemptive correction of structural inequities rather than post-hoc mitigation.

Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneous, this aggregation induces a procedural fairness failure where majority preference groups dominate reward learning while minority preferences are systematically under-represented. This work defines procedural fairness in alignment as preserving distinct preference signals during reward modeling and shows that standard RLHF violates this via preference averaging. Preference-Aware RLHF (PA-RLHF) is introduced, separating optimization across preference modes at the reward learning stage. In a controlled setting, PA-RLHF improves overall alignment accuracy from 46.9% to 67.9% and reduces the fairness gap between best and worst aligned groups from 15.9 to 9.6 percentage points. These results show that procedural fairness failures in alignment can arise from structural design choices in reward learning, even in controlled, noise-free settings, with direct implications for large language models and agentic systems, where biased reward models can compound inequities across sequential decisions.