Kairos: Toward a Byte-Level Foundation Model for Every Modality

arXiv:2610.12495 · cs.LG, cs.AI, cs.CL · Submitted 2026-09-26 · Read on arXiv

cs.LG, cs.AI, cs.CL

Submitted: 2026-09-26

Updated: 2026-09-26

Comments: 15 pages

Code: https://github.com/fabienfrfr/Kairos

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Foundation Model — 1× in this paper · explained in 3 episodes

Sources

• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Titans: Learning to Memorize at Test Time
• Longformer: The Long-Document Transformer
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
• DeepSeek-V3 Technical Report
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• PixelBytes: Catching Unified Representation for Multimodal Generation
• TPTT: Transforming Pretrained Transformers into Titans
• Gemma 3 Technical Report
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Deep Residual Learning for Image Recognition
• Training Compute-Optimal Large Language Models
• Densely Connected Convolutional Networks

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks