Mixture of Wide Experts: Sparse Routing for Interpretable Native Coordinates
cs.LG, cs.CL
Submitted: 2026-08-26
Updated: 2026-08-26
Comments: 47 pages, 14 figures. Code: https://github.com/FAIRC-Research/mowe. Models: https://huggingface.co/FAIRC
Code: https://github.com/FAIRC-Research/mowe
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language models often spread a concept across many internal dimensions, making their features difficult to interpret.
Terminology
Abstract
Language models often spread a concept across many internal dimensions, making their features difficult to interpret. Sparse autoencoders (SAEs) recover clearer features after training. We ask whether models can instead learn interpretable output coordinates directly. We introduce Mixture of Wide Experts (MoWE), a sparse feed-forward layer in which selected experts read the full representation but write to fixed, disjoint output coordinates. On a controlled task with known hidden causes, this design preserves predictive performance while localizing causes more clearly than dense and standard mixture-of-experts (MoE) models. At approximately matched compute, we train two 19-layer runs of each architecture for 10 billion tokens. MoWE achieves evaluation cross-entropy 2.417 versus 2.452 for dense, while standard MoE reaches 2.22. We measure native-coordinate readability by asking a language-model judge to describe each coordinate from passages where it is strongly active and test that description on new passages. Correcting for descriptions that also fit other coordinates gives a specificity score of 0.523 for MoWE, 0.375 for dense, and 0.268 for standard MoE. TopK SAEs trained on the same FFN updates score 0.489 for dense and 0.556 for MoWE, making native MoWE comparable to the dense SAE. In the first case study, we identify a four-coordinate causal path supporting the model's prediction of degrees after the prompt A right angle is 90. We then copy seven angle-related coordinate values into the model as it processes A rectangle has four, changing its next-token prediction from sides to degrees. In the second, one fixed coordinate set detects hazardous requests and another steers the next token toward refusal. These examples show that MoWE combines sparse conditional computation with coordinates that can be identified and reused directly.
Sources
- Network Dissection: Quantifying Interpretability of Deep Visual Representations
- Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Transcoders Find Interpretable LLM Feature Circuits
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Scaling and evaluating sparse autoencoders
- Localizing Model Behavior with Path Patching
- The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level
- Rigorously Assessing Natural Language Explanations of Neurons
- Sparse Autoencoders Do Not Find Canonical Units of Analysis
- OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
- Efficient Dictionary Learning with Switch Sparse Autoencoders
- OLMoE: Open Mixture-of-Experts Language Models
- Monet: Mixture of Monosemantic Experts for Transformers
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
- ST-MoE: Designing Stable and Transferable Sparse Expert Models
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis