MoCo: A One-Stop Shop for Model Collaboration Research

The gist The MOCO library presents a one-stop Python framework for executing, benchmarking, and comparing diverse model collaboration algorithms at scale to advance beyond single monolithic language models;

MOCO Overview

MOCO is introduced as a comprehensive Python library for model collaboration research that supports 26 diverse approaches spanning four levels of information exchange and incorporates 25 built-in evaluation datasets; It features "a wide range of 26 model collaboration algorithms, spanning four levels of collaboration defined by the level of information exchange: API-level (e.g., routing (Ong et al., 2025) and switching (Feng et al., 2025d), text-level (e.g., debate (Du et al., 2023) and cooperate (Yu et al., 2025)), logit-level (e.g., collective decoding (Liu et al., 2024a)), and weightlevel (e.g., merging (Yadav et al., 2024) and parameterspace search (Feng et al., 2025c)) MOCO provides "flexible implementations of model collaboration strategies, supporting their execution and evaluation with any amount of any LMs with any hardware setting (e.g., any amount of GPUs), democratizing model collaboration research for small models and less compute"

Model Collaboration Methods

The 26 model collaboration algorithms are categorized into four levels based on the level of information exchange across LLMs; API-level approaches aim to select the best LLM in the pool for response generation, through paradigms such as routing, cascading, and switching Text-level collaboration approaches feature exchanges of generated texts among models, through paradigms such as “debate”, “feedback”, and “discuss” Logit-level collaboration approaches involve employing and transforming the token probability distributions among multiple LLMs for collaboration Weight-level collaboration approaches utilize arithmetic and merging in the model parameter space for collaboration Specific methods include Method 1 Prompt Routing, Method 2 Nudging, Method 3 Switch Generation, and Method 26 ExPO which employs model extrapolation in the parameter space over the pool of LLMs

Evaluation and Results

MOCO comes with 25 built-in evaluation datasets (and growing), spanning reasoning, math, QA, knowledge, science, instruction following, safety, coding, computational social science, and more Extensive experiments demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average The most effective methods outperforming by up to 25.8% For instance, Weight-level is in general the most effective, achieving an average performance of 60.1 compared to the global average of 53.5 Furthermore, Model Swarms, Sparta Alignment, LLM Blender, and Agg-LM are among the best with three of them being text-level collaboration methods

Analysis and Future Directions

The paper analyzes the scaling of model collaboration systems by experimenting with scaling up the number of participating models to 2, 4, 8, and 16 LLMs It also investigates the benefit of model diversity by testing a×b settings and showing a consistent upward trend with the increase of model diversity The research highlights several key findings regarding collaborative emergence: collaborative emergence is a consistent phenomenon across various settings, with an average of 18.5% problems now solvable with model collaboration algorithms The paper concludes by envisioning MOCO as a valuable framework to spearhead a new generation of modular, bottom-up, and compositional AI systems It also addresses critical research questions such as How do we scale up model collaboration? and What are the risks of having malicious models in model collaboration systems? The impact statement emphasizes the need to study the safety of model collaboration systems: studying the impact of malicious models in decentralized model collaboration systems

Implementation Details

MOCO employs two representative model pool settings for benchmarking, Model Pool 1 featuring 3 specialized LLMs (Jiang et al., 2025c) fine-tuned on different domains of data in Tulu-v3 (Lambert et al., 2024) and Model Pool 2 featuring 3 general-purpose LLMs, specifically QWEN-2.5-7B (Yang et al., 2024), LLAMA-3.1-8B (Grattafiori et al., 2024), and OLMO-3-7B (Olmo et al., 2025) The experiment details specify default hyperparameters such as 512 max new tokens, 1024 for code generation, temperature τ = 0.7, and top-p p = 0.9 sampling for text generation The paper releases MOCO publicly at https://github.com/BunsenFeng/model collaboration and commits to providing continuous support for MOCO even after publication

Conclusion

MOCO is presented as a comprehensive toolkit that integrates 26 model collaboration algorithms, 25 evaluation datasets, and is extensible for new methods and datasets Extensive experiments demonstrate that model collaboration approaches boost participating LLMs in 61.0% of cases across diverse settings The framework enables the study of model collaboration, compositional AI systems, collaborative development, and more This work aims to turbocharge the quest for an open, modular, decentralized, and collaborative AI future

References

Austin et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021 Brahman et al. The art of saying no: Contextual noncompliance in language models. Advances in Neural Information Processing Systems, 37:49706–49748, 2024 Cai et al. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering, 2025 Chen et al. Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2023a Chen et al. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889–7901, 2023b Chiu et al. Culturalbench: A robust, diverse, and challenging cultural benchmark by human-ai culturalteaming. arXiv preprint arXiv:2410.02677, 2024 Clark et al. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018 Cobbe et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021 Du and Kaelbling Position: Compositional generative modeling: A single model is not all you need. In Forty-first International Conference on Machine Learning, 2024 Dubois et al. AlpacaEval: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039–30069, 2023 Feng et al. Nudging: Inference-time alignment of llms via guided decoding. arXiv preprint arXiv:2410.09300, 2024 Feng et al. Knowledge card: Filling llms’ knowledge gaps with plug-in specialized language models. In The Twelfth International Conference on Learning Representations, 2024a Feng et al. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14664–14690, 2024b Feng et al. When one llm drools, multi-llm collaboration rules. arXiv preprint arXiv:2502.04506, 2025a Feng et al. Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems. arXiv preprint arXiv:2502.04510, 2025b Feng et al. Model swarms: Collaborative search to adapt llm experts via swarm intelligence.

1. Model collaboration systems can solve impossible problems, as demonstrated by collaborative emergence, which is quantified as an average of 18.5% across diverse tasks like reasoning (Figure 4). This means a system composed of multiple models can achieve capabilities beyond what any single model possesses individually.

2. The system can dynamically select the best model for a query using prompt-based selection, where an LLM (QWEN-2.5-7B) is given the task description, the description of all candidate LLMs, and asked to select a subset of them for collaboration (Figure 5). This allows tailoring collaboration to specific needs rather than relying on fixed model pools.

3. The AI system can achieve superior performance on objective tasks by employing weight-level collaboration methods, as Weight-level is in general the most effective, achieving an average performance of 60.1 compared to the global average of 53.5 (Table 1). This suggests that merging or arithmetic operations on model parameters provide a more powerful mechanism for collaborative improvement than other methods.

4. The system can enhance reasoning and QA capabilities by using text-level collaboration approaches such as Multiagent Debate, which allows each model first independently generates an answer, then refine their answer based on the answers of other LLMs (Method 9). This iterative refinement process is particularly effective for math and reasoning tasks.

5. The system can improve safety performance by utilizing logit-level collaboration strategies like Logit Contrastive, which shows the highest emergence rate of 27.0% in the Safety domain (Figure 7). This suggests that manipulating token probability distributions across models is a powerful way to identify and mitigate safety risks.

6. The system can adapt its collaborative structure based on the task by using different collaboration methods; for instance, Trained router works better with specialized LMs (pool 1) than general LMs (pool 2), potentially due to the artificial hivemind phenomenon (Results section). This enables intelligent routing decisions that leverage model specialization.

7. The system can be designed for scalability by scaling up the number of participating models, as model collaboration algorithms are generally scalable: from 2 models to 16 models, there are consistent upward trends across reasoning, QA, and safety evaluation domains (Figure 2). This allows building large-scale compositional AI systems where components remain small but the overall system is powerful.

8. The system can be trained for novel skills by using Model Swarms, which collaboratively search in the model weight space to find better parameter values based on dev set performance (Method 24). This enables the discovery of new optimal parameters through a swarm intelligence approach, leading to improved performance on unseen data.

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.