Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Monarch-MoE introduces structured experts for continual instruction tuning by replacing dense LoRA factors with block-diagonal factors connected by a fixed permutation. This structure achieves blockwise gradient isolation, which improves first-task retention while significantly reducing adapter parameters. The method is practical for memory-limited hardware and supports efficient fine-tuning.