UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

Name: UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
Uploaded: 2026-10-15T09:42:58.548958+00:00
Duration: 210 s
Description: UMM-RM proposes an Upcycle-and-Merge Mixture-of-Experts Reward Model to stop reward hacking in RLHF. It copies layers from a pre-trained model to create shared and specialized experts, then merges them back into a single dense model using learnable weights. This method maintains expert diversity for robustness while keeping the efficiency of a standard reward model.

UMM-RM proposes an Upcycle-and-Merge Mixture-of-Experts Reward Model to stop reward hacking in RLHF. It copies layers from a pre-trained model to create shared and specialized experts, then merges them back into a single dense model using learnable weights. This method maintains expert diversity for robustness while keeping the efficiency of a standard reward model.

Open the full episode