Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines

arXiv:2610.03087 · cs.LG, cs.AI, cs.DB · Submitted 2026-10-02 · Read on arXiv

cs.LG, cs.AI, cs.DB

Submitted: 2026-10-02

Updated: 2026-10-02

Comments: preprint; currently under revision at VLDB'27

Code: https://github.com/datologyai/zephon

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Foundation Model — 1× in this paper · explained in 3 episodes

Sources

• LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
• Qwen3-VL Technical Report
• Qwen2.5-VL Technical Report
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Lightweight Asynchronous Snapshots for Distributed Dataflows
• Olmix: A Framework for Data Mixing Throughout LM Development
• Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• Language models scale reliably with over-training and on downstream tasks
• Gemini: A Family of Highly Capable Multimodal Models
• OLMo: Accelerating the Science of Language Models
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• DataGen: Unified Synthetic Dataset Generation via Large Language Models
• Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
• ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
• Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance
• Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines
• Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models
• RegMix: Data Mixture as Regression for Language Model Pre-training

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks