Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
cs.LG, cs.AI
Submitted: 2026-05-22
Updated: 2026-10-28
Comments: This survey has been accepted by IJCAI 2026. Results were released by 30 April 2026. This latest replacement is an extension version of survey to the previous version, which has been submitted to ACM Computing Survey for peer-review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Mixture-of-Experts (MoE) presents a naturally compatible and scalable framework for multimodal learning, demonstrating strong adaptability across diverse modalities and tasks.
Terminology
Abstract
Mixture-of-Experts (MoE) presents a naturally compatible and scalable framework for multimodal learning, demonstrating strong adaptability across diverse modalities and tasks. Despite its growing success, a comprehensive and systematic review on the MoE metho addressing multimodal challenges remains lacking. Existing surveys tend to evaluate either multimodal learning or MoE independently from method taxonomy, overlooking the unique interplay between them. This survey fills that gap by answering a central question: How does MoE effectively resolve multimodal challenges? We approach this from three key perspectives: (1) MoE as an Efficient Multimodal Engine: enabling scalable multimodal modeling by decoupling computational cost from parameter growth and mitigating modality redundancy through selective expert activation; (2) MoE as a Multimodal Representation Learner: integrating complementary multi-opinion expert knowledge to enrich alignment and interaction representations; and (3) MoE as a Multimodal Adapter: providing a modular and flexible mechanism to model imperfect data scenarios such as modality imbalance and missing modality. Through our extensive literature review, we identify critical research gaps, including interpretable routing, expert communication, modality integration, and lifelong multimodal learning. We position this survey as a foundation for future research toward interpretable and sustainable multimodal Mixture-of-Experts system.
Sources
- MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- Upcycling Large Language Models into Mixture of Experts
- Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
- Efficient Multimodal Large Language Models: A Survey
- M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecture
- R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
- MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
- MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
- $\gamma-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models
- From Efficient Multimodal Models to World Models: A Survey
- UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths
- From Sparse to Soft Mixtures of Experts
- Ctrl-G: Controlling Structural and Interaction Exposure in Knowledge Graph Augmented LLM Reasoning
- I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
- Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
- Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate
- SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis