RA-MoE: Routing-Aligned Fine-Tuning for Multilingual Adaptation of Mixture-of-Experts Models
RA-MoE is a three-stage method to improve non-English performance in Mixture-of-Experts (MoE) models by aligning their routing structure with English task patterns. It identifies language-universal expertise in middle layers and uses a specific loss function during fine-tuning to guide the model's routing toward these relevant experts for target languages.