SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
SMoES introduces dynamic soft modality scores to guide expert routing in Vision-Language Models (MoE-VLMs). It uses these scores to create expert bins based on layer-dependent fusion patterns and mutual information regularization. This enables modality-aware specialization, leading to significant gains in both model effectiveness and efficiency, such as a 56.1% reduction in communication overhead.