Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models
Mixture-of-Control (MoC) is a lightweight state-based fine-tuning method that improves transformer adaptation by integrating local and global control signals. MoC treats block states as experts in a mixture-of-experts process, allowing efficient cross-block communication without high computational overhead. Experiments show MoC consistently outperforms existing methods while maintaining comparable memory and compute efficiency across various benchmarks.