Universal Redundancies in Time Series Foundation Models
cs.LG, stat.ML
Submitted: 2026-02-02
Updated: 2026-08-30
Code: https://github.com/abao1999/tsfm-lens
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning.
Terminology
Abstract
Time Series Foundation Models (TSFMs) leverage extensive pretraining to accurately predict unseen time series during inference, without the need for task-specific fine-tuning. Through large-scale evaluations on standard benchmarks, we find that leading transformer-based TSFMs exhibit redundant components in their intermediate layers. We introduce a set of tools for mechanistic interpretability of TSFMs, including ablations of specific components and direct logit attribution on the residual stream. Our findings are consistent across several leading TSFMs with diverse architectures, and across a diverse set of real-world and synthetic time-series datasets. We discover that all models in our study are robust to ablations of entire layers. Furthermore, we develop a theoretical framework framing transformers as kernel regressors, motivating a purely intrinsic strategy for ablating heads based on the stable rank of the per-head projection matrices. Using this approach, we uncover the specific heads responsible for degenerate phenomena widely observed in TSFMs, such as parroting of motifs from the context and seasonality bias. Our study sheds light on the universal properties of this emerging class of architectures for continuous-time sequence modeling.
Sources
- Conversational Time Series Foundation Models: Towards Explainable and Effective Forecasting
- GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
- Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs
- This Time is Different: An Observability Perspective on Time Series Foundation Models
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Constructing Efficient Fact-Storing MLPs for Transformers
- TimeGPT-1
- How Do LLMs Use Their Depth?
- Language Models Implement Simple Word2Vec-style Vector Arithmetic
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
- How much do language models memorize?
- Understanding Factual Recall in Transformers via Associative Memories
- Panda: A pretrained forecast model for chaotic dynamics
- Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts
- Test-time regression: a unifying framework for designing sequence models with associative memory
- When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
- Understanding the Implicit Biases of Design Choices for Time Series Foundation Models
- Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
- AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Prediction
- Zero-shot forecasting of chaotic systems
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks