TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting
cs.LG, cs.AI, stat.ML
Submitted: 2025-10-29
Updated: 2026-11-02
Code: https://github.com/automl/TempoPFN
License: http://creativecommons.org/licenses/by/4.0/
The gist: Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility…
Terminology
Abstract
Foundation models for zero-shot time series forecasting face challenges in efficient long-horizon prediction and reproducibility, with existing synthetic-only approaches underperforming on challenging benchmarks. This paper presents TempoPFN, a univariate time series foundation model based on linear Recurrent Neural Networks (RNNs) pre-trained exclusively on synthetic data. The model uses a GatedDeltaProduct architecture with state-weaving for fully parallelizable training across sequence lengths, eliminating the need for windowing or summarization techniques while maintaining robust temporal state-tracking. Our comprehensive synthetic data pipeline unifies diverse generators, including stochastic differential equations, Gaussian processes, and audio synthesis, with novel augmentations. In zero-shot evaluations on the Gift-Eval, fev-bench and Chronos-ZS benchmarks, TempoPFN achieves top-tier competitive performance, surpassing the majority of models trained on real-world data, while leveraging fully parallelizable training and inference. We open-source our complete data generation pipeline and training code, providing a reproducible foundation for future research.
Sources
- Linear Attention for Efficient Bidirectional Sequence Modeling
- TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning
- Titans: Learning to Memorize at Test Time
- Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models
- FlowState: Sampling-Rate-Equivariant Time-Series Forecasting
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Gaussian Error Linear Units (GELUs)
- Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- Beyond Data Scarcity: A Frequency-Driven Framework for Zero-Shot Forecasting
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States
- LLaMA: Open and Efficient Foundation Language Models
- CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis