UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

The gist The UMM-RM framework proposes an Upcycle-and-Merge Mixture-of-Experts Reward Model to mitigate reward hacking in Reinforcement Learning from Human Feedback by combining expert diversity with parameter consolidation.

Motivation and Problem

Reward models (RMs) in RLHF are susceptible to exploitation by policy models through biases or spurious correlations, resulting in reward hacking where RM scores increase during training while alignment with human preferences deteriorates, a problem that is further exacerbated under distribution shift Reward hacking is commonly referred to as reward hacking <ref:2512.00724#pg7>. During RLHF, the policy continuously pushes outputs into regions where the reward model provides weak supervision, further leveraging and amplifying uncertainties and systematic biases, ultimately causing a decoupling between the reward and true quality and resulting in a decline in generation quality <ref:2512.00724#pg7>. Prior work has pursued multiple directions to mitigate reward hacking <ref:2512.00724#pg8>. However, these approaches incur significant computational and resource costs at inference, as multiple reward models must be queried <ref:2512.00724#pg3>.

UMM-RM Architecture

The UMM-RM first upscales the feed-forward layers of a dense backbone into a mixture-of-experts (MoE) reward model with shared experts <ref:2512.00724#pg4>. The shared experts are always activated to capture instruction-agnostic preference signals, while the remaining experts model fine-grained preferences across instructions or task regimes <ref:2512.00724#pg4>. After training, the experts are consolidated into a single dense RM via learnable merging weights <ref:2512.00724#pg4>. This design retains the robustness and exploitation resistance provided by expert diversity while avoiding the inference overhead of MoE architectures or explicit ensembles <ref:2512.00724#pg3>.

Training and Merging Strategy

The training process involves decomposing representations into “general + specialized” components, where the shared expert provides a stable, low-variance baseline <ref:2512.00724#pg4>. The gating weight of the shared expert is fixed by a hyperparameter α ∈ [0, 1] and is activated for all tokens <ref:2512.00724#pg4>. For the standard experts, the router produces routing logits r(l)e,t <ref:2512.00724#pg4>. After training, the MoE layers are merged in parameter space through learnable weighted averaging to obtain standard feedforward network (FFN) layers, thereby constructing a dense reward model <ref:2512.00724#pg4>. Specifically, the parameters of the merged dense FFN at layer l are given by Θ(l)merge = α Θ(l)0 + K X−1 e=1 g¯(l)e Θ(l)e <ref:2512.00724#pg4>. This merging step reduces the high-variance reward responses introduced by individual misaligned experts and weakens the reliance on spurious features, thus significantly mitigating reward hacking during RLHF, while maintaining the same parameter scale and inference cost as the dense reward model <ref:2512.00724#pg3>.

Experimental Validation

Experiments across multiple base models and preference datasets show that UMM-RM improves accuracy on preference data, reduces reward hacking during PPO training, and yields more stable preference alignment <ref:2512.00724#pg4>. Evaluation on TinyLlama-1.1B showed that the dense RM’s proxy score increases monotonically with training, whereas UMM-RM exhibits a steadier increase in proxy score and a generally consistent improvement in gold score <ref:2512.00724#pg6>. Furthermore, enabling more experts (4/6) suppresses reward hacking more effectively than enabling fewer (2) <ref:2512.00724#pg6>. In end-to-end generation quality assessments, UMM-RM achieves alignment performance comparable to ensemble and MoE reward models while maintaining computational costs close to the dense reward model <ref:2512.00724#pg4>.

Conclusion

Overall, UMM-RM yields higher win rates than both the dense RM and SFT, suggesting it steers PPO toward responses aligned with human preferences rather than simply maximizing reward scores <ref:2512.00724#pg8>. The upcycle-and-merge shared-expert design provides more effective reward supervision than simple ensembling <ref:2512.00724#pg8>. Overall, UMM-RM offers a practical and efficient approach for constructing reliable reward models.

1. The UMM-RM framework mitigates reward hacking by decomposing preference modeling into general + specialized components, where the shared expert provides a stable, low-variance baseline, and the routed experts add expressive capacity. This allows the model to capture both instruction-agnostic preference signals and fine-grained preferences across instructions or task regimes.

2. The upcycle-and-merge strategy ensures computational efficiency by consolidating the MoE back into a single dense reward model, which retains the robustness of expert diversity while avoiding the overhead of explicit ensembles or MoE inference. This results in higher throughput and lower resource consumption at deployment.

3. The learnable merging step suppresses overfitting by attenuating extreme weights from individual experts, which leads to reward smoothing and reduces the policy’s tendency to overexploit individual experts or local biases. This makes the reward function less sensitive to local input perturbations, thereby compressing the exploitable space available to policy models.

4. By increasing activated experts, UMM-RM improves stability: experiments show that increasing the number of activated experts can substantially improve the stability and robustness of reward evaluation. Furthermore, a moderate shared-expert weight coefficient (0.5) is found to be optimal for PPO training, suggesting that moderate sharing stabilizes the reward function and mitigates noise.