Mixture-of-Models: Distributed Inference over Heterogeneous LLMs as Model-Level Mixture-of-Experts
cs.AI, cs.LG, cs.MA, cs.SY, eess.SY
Submitted: 2026-01-23
Updated: 2026-10-31
Project page: https://abehrouz.github.io/files/NL.pdf
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Titans: Learning to Memorize at Test Time
- End-to-End Test-Time Training for Long Context
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
- Generative Agents: Interactive Simulacra of Human Behavior
- From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
- Why Do Multi-Agent LLM Systems Fail?
- Lost in the Middle: How Language Models Use Long Contexts
- Towards Understanding Sycophancy in Language Models
- DarkBench: Benchmarking Dark Patterns in Large Language Models
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
- Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
- STaR: Bootstrapping Reasoning With Reasoning
- Training Verifiers to Solve Math Word Problems
- Let's Verify Step by Step
- ReAct: Synergizing Reasoning and Acting in Language Models
- Attention Is All You Need
- The Curse of Recursion: Training on Generated Data Makes Models Forget
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents