Fractional State Space Transition for Long Sequence Modeling
cs.LG, cs.CL
Submitted: 2026-09-28
Updated: 2026-09-28
Comments: NeurIPS 2026 (Oral)
Code: https://github.com/anasiri/frac-ssm
License: http://creativecommons.org/licenses/by/4.0/
The gist: State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance.
Terminology
Abstract
State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
Sources
- Just read twice: closing the recall gap for recurrent language models
- An Efficient Exponential Sum Approximation of Power-Law Kernels for Solving Fractional Differential Equation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Neural Fractional Differential Equations
- ProT-GFDM: A Generative Fractional Diffusion Model for Protein Generation
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
- Decoupled Weight Decay Regularization
- Retentive Network: A Successor to Transformer for Large Language Models
- Kimi Linear: An Expressive, Efficient Attention Architecture
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Qwen3 Technical Report
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks