Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?

arXiv:2507.02907 · cs.LG · Submitted 2025-06-24 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Scaling Transformers for Time Series Forecasting".

Jane: The gist The work empirically examines whether pretrained large-scale time series models (LSTSMs) trained on diverse datasets can outperform traditional non-pretrained small-scale transformers in forecasting tasks.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, this paper is called "Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?" and the authors are Sanjay Chakrabortya, Ibrahim Delibasoglub, and Frerik Heintzc. Jane It’s a pretty straightforward title for what they’re doing, which is testing whether these massive pre-trained models actually beat the smaller transformers we usually see in time series work. Lu The main idea here is to empirically examine if large-scale time series models, or LSTSMs, trained on lots of data, can outperform traditional small-scale transformers when forecasting. Meng So we’re looking at the performance difference between these two types of AI models across different tasks.

Tom: Exactly. What’s interesting is they aren't just looking at one thing; they are comparing state-of-the-art pre-trained universal time series models, like Moirai or TimeGPT, against conventional transformers that haven't been pre-trained for this specific job yet. Jane It’s important to understand that the whole point is to find out where the pretraining gives an advantage and where starting from scratch with a smaller model might actually be better.

Lu: The paper sets up a comparison across several things, specifically looking at accuracy, computational efficiency, and interpretability when comparing these models. Tom That makes sense. We’re not just chasing the highest score; we’re checking if that high score is earned by a model that’s actually useful in the real world.

Jane: It seems like they are setting up a framework to decide which type of time series model is better for different jobs, whether it's long-term prediction or something more immediate. Meng And they’re testing this across diverse datasets, which is good because you don't want a model that only works well on one specific type of data.

Lalam: I see them looking at how these models handle things like missing data and if they can even manage multivariate time series, which is when you have multiple different variables interacting. Tom Right, so they’re checking the breadth of capability too, not just the raw number on a single test set.

Jane: And that’s what we want to hear—a model that shows strong generalization across different types of time series problems, not just one perfect result on one dataset.

The paper's summary: Tom: Okay, moving into the actual research, the paper summarizes their findings by focusing on how these large-scale models handle sequence processing and prediction. Jane They break down the difference between how traditional transformers process everything at once versus how these large-scale models operate autoregressively. Lu The core distinction they make is that standard transformers attend to all time steps simultaneously, which is great for classification where you need context from the entire sequence, but large-scale models focus on predicting future time steps based only on past observations.

Tom: They use a mathematical notation to show this difference: A equals softmax QKT√dk versus the causal prediction of xˆT +one equals fLSTSM(xone xtwo <ref:2507.02907#pg3>..., xT) thirty-six four <ref:2507.02907#pg2>. Jane That’s a big conceptual jump for a listener who isn't reading the math—it means one is looking at the whole movie at once while the other is just predicting what happens next based on what’s already happened.

Meng: For forecasting, that autoregressive nature is key because in real-world scenarios, you usually don't know what happens next; you can only use past data to make a guess. Tom And they use this distinction to argue why large-scale models are especially effective for forecasting and anomaly detection, because they can flag weird values by comparing their prediction to the actual value.

Lu: They also look at efficiency, which is a huge point since these models are notoriously heavy. They compare the quadratic complexity of self-attention in Transformers against the more efficient scaling of LSTSMs for autoregressive tasks. Jane So they’re saying that for a forecasting task, you might get a better trade-off between power and speed with an LSTSM.

Tom: And they show that pretraining on large amounts of data gives these models a massive advantage in generalization across different domains, which is what they call scaling to large datasets one <ref:2507.02907#pg3>. Meng That’s the transfer learning angle—the idea that what you learn on one huge dataset helps it perform better when it sees something slightly different.

Lalam: I think the main implication here is that for forecasting, if you have access to massive pre-training data, leaning into these large-scale models seems like a solid strategy over just using a small transformer trained only on your specific data.

The paper's improvements: Tom: Now let’s talk about what they suggest we should actually do with this research. They highlight several areas where the approach can be improved or applied more effectively. Jane One big suggestion is to look at how these models can handle missing data, which is a real headache in real time series. Lu The paper points out that large-scale models can infer missing points contextually, and they explicitly handle those gaps instead of just ignoring them.

Tom: That’s a practical improvement for any forecasting system; if you have incomplete data, the model should be able to fill in the blanks intelligently rather than throwing an error. Meng From my side, I think that contextual inference is crucial for deploying these models because real-world data is rarely perfect.

Jane: They also discuss how pretraining can stabilize the model’s performance when you fine-tune it on a new, specific domain, like traffic data or something entirely different. Lu They found evidence suggesting that models benefiting from pretraining on Era5 generally exhibit more stable alpha values within the optimal range when fine-tuned on new domains like ETTh1.

Tom: So it’s not just about the initial training; it’s about how that pretraining translates into better behavior when you adapt it for your specific use case. Meng That makes sense from an engineering standpoint—we need reliable performance after we spend time adapting the model.

Jane: And they touch on flexibility regarding prediction horizons. They suggest that transformers, because they attend to everything at once, are better suited for multi-step forecasting where you want to predict several future time steps simultaneously using bidirectional attention. Tom So it’s a trade-off: autoregressive models for simple next-step predictions versus simultaneous learning across the whole sequence.

Lu: That’s a key architectural direction they are pushing, suggesting that we should be looking at how to combine the strengths of both architectures for different prediction needs.

Conclusion: Tom: So, to wrap up this paper on "Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?", the main conclusion is that pretrained large-scale transformer models, like LLM4TS FS and Timer XL, significantly outperform smaller, task-specific models by minimizing expected loss functions. Jane They do this across diverse datasets and prediction horizons when looking at metrics like MSE and MAE.

Lu: Essentially, the paper confirms that pretraining offers a clear advantage for multivariate long-term time series forecasting compared to training from scratch with smaller architectures. Meng From my experience in production, it means if you need high accuracy on long sequences, leveraging that pre-trained knowledge seems like the way to go.

Lalam: I’m just confident in this result because the evidence points toward these LSTSMs being better at capturing complex temporal patterns when trained broadly. Tom And while models trained from scratch, like Moment and Timer, are still competitive, they don't quite match that level of performance across the board.

Jane: The implication for people listening is that if you’re working on a forecasting problem with a lot of historical data and you want the best possible prediction accuracy across different scenarios, using these large-scale pre-trained models seems like the most reliable path right now. Meng And remember, they also pointed out that their method doesn't offer full interpretability right now, which is something we have to keep in mind when deploying them.

Lu: That’s true. The paper lays out the current state of things very clearly and gives us a solid foundation for future work in model adaptation. Tom It’s a comprehensive study on the strengths and weaknesses of LSTSMs versus small-scale transformers, highlighting where we should focus our efforts next.

Jane: That covers what this paper does, from the title to the authors to what it suggests for the future of time series forecasting.

Lalam: I think this comparison between pre-trained models and small alternatives is a really important piece of work in understanding how scaling impacts model performance in this domain.

Sanjay Chakrabortya, Ibrahim Delibasoglub, Frerik Heintzc

Department of Computer and Information Science (IDA), REAL, AIICS, Linköping University, Sweden

cs.LG

Submitted: 2025-06-24

Updated: 2025-06-24

Journal ref: Artificial Intelligence Review, 59(62), 2026

DOI: 10.1007/s10462-025-11481-7

Code: https://github.com/UVA-MLSys/SA-Timeseries

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 80/100

The gist: The gist The work empirically examines whether pretrained large-scale time series models (LSTSMs) trained on diverse datasets can outperform traditional non-pretrained small-scale transformers in

Key concepts

Large-scale Time Series Models (LSTSMs): These are massive time series models inspired by Large Language Models. They leverage huge datasets and extensive pretraining to capture complex temporal patterns. Unlike standard transformers, they process data autoregressively, predicting future steps based only on past observations, making them highly effective for forecasting.
Transformers: Transformers are a class of neural networks that use an attention mechanism to process sequential data. They can look at all time steps in a sequence simultaneously. While powerful for context capture, they can be computationally expensive because their self-attention mechanism scales quadratically with the sequence length.
Autoregressive Modeling: This is a method where a model predicts the next element in a sequence based on all the elements that came before it. Large-scale models use this approach to forecast time series, as they only need past data to predict future values. This contrasts with models that might look at both past and future information at once.
Pretraining vs. Training from Scratch: Pretraining involves training a model on vast, diverse datasets before fine-tuning it for a specific task. The research found that pretrained LSTSMs consistently outperform smaller models trained only on the target time series data from scratch. This demonstrates the value of transfer learning in improving forecasting performance.

Terminology

Transformer — 3× in this paper · explained in 2 episodes
Time Series Forecasting — 2× in this paper · explained in 4 episodes
Accuracy — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Interpretability — 1× in this paper · explained in 15 episodes

Summary

The gist The work empirically examines whether pretrained large-scale time series models (LSTSMs) trained on diverse datasets can outperform traditional non-pretrained small-scale transformers in forecasting tasks.

Background

Time series analysis is a crucial technique in data science, enabling insights into temporal data patterns <ref:2507.02907#pg2> Time series can be classified as either univariate or multivariate [44] <ref:2507.02907#pg4>. Time series forecasting is further categorized into short-term and long-term forecasting based on the prediction horizon <ref:2507.02907#pg2> Both require effective modeling of temporal dependencies to ensure accurate predictions >. Large-scale Time Series Models (LSTSMs) have surpassed traditional transformers in various tasks due to their scalability, extensive pretraining, security, and adaptive capabilities [1] <ref:2507.02907#pg4>. Unlike standard transformers, which process fixed-length sequences with selfattention, large-scale models leverage massive datasets, increased parameter counts, and architectural innovations such as sparse attention and retrieval-augmented generation >.

Architectural Differences

The distinction between Transformers and Large-scale Time Series Models (LSTSMs) lies in their processing of sequential data <ref:2507.02907#pg3> In Transformers, the attention mechanism processes all time steps simultaneously, allowing the model to capture both past and future information <ref:2507.02907#pg4>. In contrast, large-scale models inspired by large language models (LLMs), like GPT, process the time series data in an autoregressive manner, predicting future time steps based on past observations only <ref:2507.02907#pg5>. This mathematical distinction is shown as A = softmax QKT√dk whereas large-scale models predict future values in a causal manner as xˆT +1 = fLSTSM(x1, x2..., xT) <ref:2507.02907#pg4>.

Handling Sequential Data and Tasks

Transformers typically attend to all time steps in a sequence simultaneously, processing data from both the past and future <ref:2507.02907#pg5> This makes them suitable for tasks like time series classification, where context from the entire sequence is important <ref:2507.02907#pg4>. On the other hand, large-scale models operate autoregressively, utilizing only past time steps to predict future values <ref:2507.02907#pg5> This is particularly useful for tasks like time series forecasting, where only past data is relevant <ref:2507.02907#pg4>. For anomaly detection, large-scale models can flag anomalous values based on prediction errors by comparing predicted values to actual ones <ref:2507.02907#pg4>

Model Efficiency and Complexity

Transformers are highly expressive but computationally expensive due to the quadratic complexity of self-attention <ref:2507.02907#pg5> For long time series, the attention matrix grows as O(T2), which becomes computationally prohibitive for very large sequences <ref:2507.02907#pg4>. LSTSMs, however, are more efficient for autoregressive tasks, as they only attend to past time steps for prediction <ref:2507.02907#pg5> The computational cost for Transformers scales with the entire sequence, whereas LSTSMs scale with the autoregressive nature of prediction, processing one step at a time <ref:2507.02907#pg4>.

Empirical Analysis and Findings

The study evaluates SOTA pre-trained Large-scale Time Series Models (LSTSMs) against small-scale transformers across long-term and short-term forecasting tasks <ref:2507.02907#pg2> Figure 4 demonstrates a performance comparison of the experimented state-of-the-art (SOTA) large-scale and small-scale language models across various benchmark datasets for long-term and short-term forecasting tasks, respectively <ref:2507.02907#pg4>. The results highlight scenarios where pretraining offers advantages and where simpler models remain competitive >. Theorem 1 states that in multivariate long-term time series forecasting, pretrained and fine-tuned models minimize the expected loss E[L(f(X), Y)] more effectively than models trained from scratch <ref:2507.02907#pg9>. Theorem 3 confirms that in multivariate short-term traffic forecasting, Timer XL consistently achieves the lowest error values in both MSE and MAE across varying prediction horizons and datasets <ref:2507.02907#pg9>.

Conclusion

Pretrained large-scale transformer models, such as LLM4TS FS and Timer XL, significantly outperform smaller, task-specific models by minimizing expected loss functions—particularly MSE and MAE—across diverse datasets and prediction horizons <ref:2507.02907#pg9>. Their superior capacity for temporal pattern recognition, effective feature extraction via attention mechanisms, and enhanced generalization through transfer learning contribute to their state-of-the-art forecasting performance >. Large-scale models trained from scratch, such as Moment and Timer, also exhibit competitive results <ref:2507.02907#pg9>. This study aids future researchers in making informed decisions regarding model adaptation for various time series tasks >. The paper's main contributions are that it has examined the strengths and weaknesses of LSTSMs in comparison to small-scale time series transformers, highlighting key opportunities where LSTSMs can be valuable and impactful <ref:2507.02907#pg4>. It also explored recent advancements in large-scale models relevant to time series forecasting, with a primary focus on multivariate time series forecasting <ref:2507.02907#pg4>. Finally, it is a complete study to analyze and compare a set of state-of-the-art (SOTA) Long-Term Sequence Learning Models (LSTSMs) with small-scale transformers for time series forecasting tasks, focusing on computational complexity, architectural differences, and performance quality <ref:2507.02907#pg4>. The limitation noted is the lack of analysis on model interpretability and computational efficiency during inference for real-time deployment scenarios >. Future work will explore more flexible approaches for time series analysis by incorporating advanced LSTSMs <ref:2507.02907#pg9>.

Improvements for AI systems

textbfPretrained LSTSM Performance Enhancement for Long-Term Forecasting: Theorem 1 realization by LLM4TS FS or Timer XL (Long-term): The model can minimize the expected loss E[L(f(X), Y)] more effectively than models trained from scratch, specifically demonstrating superior generalization across diverse datasets.
textbfModel Selection Strategy for Task Complexity: Dynamic Model Switching: Systems can dynamically select between LSTSMs and small-scale transformers based on the required horizon; for instance, using LLM4TS FS for long-term forecasting while employing Timer XL for high-frequency short-term traffic prediction.
textbfEnhanced Anomaly Detection Capability: Contextual Outlier Flagging: The system can perform anomaly detection by leveraging the prediction errors by comparing predicted values to actual ones, as noted in the text regarding large-scale models.
textbfImproved Training Efficiency via Transfer Learning: Knowledge Transfer Optimization: Models can utilize transfer learning, as evidenced by the analysis showing that models benefiting from pretraining on Era5 generally exhibit more stable α values within the optimal range, leading to improved layer quality and generalization when fine-tuned on new domains like ETTh1.
textbfAdaptive Forecasting Horizon Handling: Multi-step Prediction Flexibility: The system can handle multi-step forecasting, as Transformers are suitable for multi-step forecasting tasks, where the model can predict multiple future time steps simultaneously using bidirectional attention.

Abstract

Large pre-trained models have demonstrated remarkable capabilities across domains, but their effectiveness in time series forecasting remains understudied. This work empirically examines whether pre-trained large-scale time series models (LSTSMs) trained on diverse datasets can outperform traditional non-pretrained small-scale transformers in forecasting tasks. We analyze state-of-the-art (SOTA) pre-trained universal time series models (e.g., Moirai, TimeGPT) alongside conventional transformers, evaluating accuracy, computational efficiency, and interpretability across multiple benchmarks. Our findings reveal the strengths and limitations of pre-trained LSTSMs, providing insights into their suitability for time series tasks compared to task-specific small-scale architectures. The results highlight scenarios where pretraining offers advantages and where simpler models remain competitive.