Always Learning, Always Mixing: Efficient and Simple Data Mixing All the Time
cs.CL, cs.AI, cs.LG
Submitted: 2026-05-13
Updated: 2026-10-28
Comments: NeurIPS 2026 camera ready
Code: https://github.com/michahu/on-policy-mix
License: http://creativecommons.org/licenses/by/4.0/
The gist: Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training.
Terminology
Abstract
Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key determinant of model quality; in continual learning and adaptation, it governs what is retained and acquired. Yet existing data mixing methods address only one phase of this lifecycle at a time: some require smaller proxy models tied to a single training phase, others assume a fixed domain set, and continual learning lacks principled guidance altogether. We argue that data mixing is fundamentally an online decision making problem---one that recurs throughout training and demands a single, unified solution. We introduce OP-Mix (On-Policy Mix), a data mixing algorithm that operates across pretraining, midtraining, and instruction tuning. Our main insight is that candidate data mixtures can be cheaply simulated by interpolating between low-rank adapters trained directly on the current model, eliminating separate proxy models and ensuring the search is always grounded in the model's actual learning dynamics. Across the language model training lifecycle, OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of the baselines. In pretraining, OP-Mix improves upon training without mixing by 6.3% in average perplexity. For continual learning, OP-Mix matches the performance of both retraining and on-policy distillation while using 66% and 95% less overall compute, respectively. OP-Mix suggests a different view of language model training: not a sequence of distinct phases, but a single continuous process of learning from data.
Sources
- Olmix: A Framework for Data Mixing Throughout LM Development
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- The Llama 3 Herd of Models
- Training Compute-Optimal Large Language Models
- Midtraining Bridges Pretraining and Posttraining Distributions
- 2 OLMo 2 Furious
- Progressive Neural Networks
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Self-Distillation Enables Continual Learning
- Merge to Mix: Mixing Datasets via Model Merging
- Gemma 3 Technical Report
- MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
- Qwen2 Technical Report
- Qwen3 Technical Report
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data