Scaling Laws for Multimodal Data Mixtures

arXiv:2610.14423 · cs.CL, cs.LG · Submitted 2026-10-10 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Terminology

• Scaling Laws — 1× in this paper · explained in 4 episodes

Sources

• LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
• Multilingual Routing in Mixture-of-Experts
• Broken Neural Scaling Laws
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Emu3.5: Native Multimodal Models are World Learners
• The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
• Scaling Laws for Autoregressive Generative Modeling
• propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale
• Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
• Scaling Laws for Neural Language Models
• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
• Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
• MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
• Voxtral
• ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality
• Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens
• MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering