When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters

arXiv:2607.04919 · cs.LG · Submitted 2026-07-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters".

Paper summary: Tom: So to recap, this paper, When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters, provides a systematic way to look at when you should invest in deploying a time series foundation model versus using classical methods like XGBoost. The thesis is that foundation models are better than classical methods on fifteen of thirty benchmark datasets across various training set sizes, justifying GPU deployment there even with small amounts of data.

Jane: What’s the big picture for us? It’s about giving practitioners a clear framework to decide before they commit to the full infrastructure investment by showing them exactly when that investment pays off based on the amount of data they have and the nature of their time series.

Lu: The analysis compares zero-shot foundation models and those fine-tuned with LoRA against classical baselines—Naive, ETS, ARIMA, and XGBoost—across six different training set sizes from two percent up to one hundred percent. This comparison helps us understand the scaling curve framework where we see how error changes as the training data volume increases.

Meng: I’m curious about the fine-tuning part. Does that LoRA fine-tuning give them a big boost, or is it just adding complexity without much gain in these scenarios?

Lalam: The paper specifically examines that relationship: when fine-tuned via LoRA, the error might decrease with more data on series with clear periodicity, but it can actually increase due to overfitting on shorter series. That’s a crucial caveat for anyone thinking about fine-tuning.

Jane: So, we're seeing this mix of zero-shot wins and conditional fine-tuning benefits depending on the dataset structure. It seems the performance isn't one single thing but depends on the relationship between the model type, how much data you feed it, and what kind of time series you are dealing with.

Tom: Exactly. And when we look at those datasets where classical methods beat zero-shot models with as little as two percent of data, that’s an early break-even point where classical methods start to get competitive very quickly.

Lu: That suggests that for those early break-even cases, you might only need a small amount of labeled or task-specific data before you see a meaningful improvement over the zero-shot model.

Meng: For my perspective on production, if we’re in an early break-even scenario, that means we can probably deploy something classical sooner because the foundation model isn't giving us enough leverage yet.

Lalam: The paper also gives us a specific rule for when no training is even necessary: if you have less than seven hundred samples and the seasonality is non-negligible, the zero-shot foundation model consistently favors you >

Conclusion: Jane: So, wrapping up this analysis of When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters, what does it actually mean for someone who isn't deep in the math?

Tom: Simply put, this paper gives you a practical guide. It tells you exactly which datasets are good candidates for deploying a foundation model right now because they show that the zero-shot performance is already strong enough to justify the GPU cost on fifteen of thirty cases.

Lu: The authors identified that these datasets fall into four categories: FM-dominant, early break-even, mid break-even, and late break-even based on how much training data classical methods need to catch up.

Meng: If you’re in a late break-even case, it signals that you might actually need to commit a lot of your time series data just to get the classical approach performing as well as the zero-shot model is already doing out of the box.

Lalam: The most important thing I see is that there’s an empirical rule for when you don't even need training at all—if you have less than seven hundred samples and seasonality is present, you should probably go with the zero-shot foundation model.

Jane: So, the implication is that instead of guessing which model to use, practitioners can use this framework to decide whether they need to bother training a foundation model or if they can just run something classical for better efficiency in certain situations.

Nicholas Tan Jerome, Frank Simon

Karlsruhe Institute of Technology (KIT)

cs.LG

Submitted: 2026-07-06

Updated: 2026-11-18

Comments: 8 pages, 4 figures, 3 tables. Accepted at the IEEE ICDM 2026 Workshop on Large Models for Time Series Data Mining (LM4TED). Code: https://github.com/nicolaisi/fm_breakeven

Code: https://github.com/nicolaisi/fm-breakeven

License: http://creativecommons.org/licenses/by/4.0/

The gist: Foundation models outperform classical methods at every evaluated training fraction on 15 of 30 datasets—GPU deployment is unconditionally justified on these regardless of data volume The paper…

Key concepts

Break-Even Classification Framework
A system used to categorize datasets based on when classical methods first match or beat zero-shot foundation models. It sorts datasets into four types: FM-dominant, early break-even (classical beats FM with 2% data), mid break-even (classical needs about 10% data), and late break-even (classical needs 20% to 100% data).
FM-dominant
Datasets where the foundation model consistently wins, meaning it outperforms classical methods regardless of how much training data is used. This indicates that GPU deployment is unconditionally justified on these specific datasets because the FM always provides superior performance.
Zero-Shot vs. Fine-Tuning
Zero-shot models are tested without any fine-tuning, while LoRA fine-tuned models are compared against classical baselines. The analysis shows zero-shot foundation models already outperform fully trained classical methods on 19 out of 30 datasets, suggesting initial deployment is valuable.
Training Fraction
This refers to the percentage of available training data used when testing model performance. The paper systematically tests different training fractions (from 2% up to 100%) to pinpoint the exact data volume required for classical methods to overcome the foundation model's initial advantage.

Terminology

Summary

Foundation models outperform classical methods at every evaluated training fraction on 15 of 30 datasets—GPU deployment is unconditionally justified on these regardless of data volume The paper provides a systematic break-even analysis answering when the investment in deploying a time series foundation model pays off This analysis compares zero-shot and LoRA fine-tuned foundation models against classical baselines across 30 benchmark datasets and six training set sizes The findings help practitioners make the FM-versus-classical decision before committing to full infrastructure. p. 0

Where the paper says it

Across 30 benchmark datasets, we compare zero-shot and LoRA fine-tuned foundation models (Chronos, Moirai, LagLlama) against classical baselines (Naive, ETS, ARIMA, XGBoost) at six training set sizes from 2% to 100% of available data. Foundation models outperform classical methods at every evaluated training fraction on 15 of 30 datasets—GPU deployment is unconditionally justified on these regardless of data volume.

Break-Even Classification Framework

The researchers introduce a framework based on three primary patterns observed across the 30 datasets These patterns are classified as FM-dominant, early break-even, mid break-even, and late break-even. The classification is based on the smallest training fraction at which any classical method first matches or beats FM zero-shot. p. 2

Where the paper says it

We determine break-even empirically from the discrete fraction grid rather than by fitting parametric curves, which avoids extrapolation artefacts on small datasets. We partition datasets into four categories based on the smallest training fraction at which any classical method first matches or beats FM zero-shot: FM-dominant (no break-even observed), Early (≤5% of data), Mid (10%), and Late (≥20%).

The four categories are defined as follows

  1. FM-dominant (n∗ = 0) represents datasets where the foundation model always wins. p. 1

    Where the paper says it

    We define the break-even point n ∗ as the smallest training set size where the classical method matches or exceeds the best foundation model: n ∗ = min{n : Error(fCL, n) ≤ Error(fFM)} (1) where fFM is evaluated in zero-shot mode. If no such n exists within the available data range, we say n ∗ = 0 (foundation model always wins, FM-dominant) or n ∗ = ∞ (classical method always wins even from minimal data).

  2. Early break-even occurs when classical methods surpass zero-shot foundation models with as little as 2% of data. p. 4

    Where the paper says it

    Pattern 2: Early Break-Even. On 6 datasets (AusElectricity, ETTh2, FRED-MD, Saugeen, Solar-10min, Traffic), classical methods surpass zero-shot foundation models with as little as 2% of training data (21–2,768 samples).

  3. Mid break-even is observed on one dataset where classical methods require approximately 10% of training data. p. 4

    Where the paper says it

    Pattern 3: Mid Break-Even. On 1 dataset (Weather), classical methods require ≈10% of training data (n ∗ = 3,161 samples).

  4. Late break-even occurs when classical methods need substantial data, ranging from 20% to 100% of available data. p. 4

    Where the paper says it

    On 6 datasets, classical methods need only 2% of available data to catch up (21–2,768 samples). The remaining 9 datasets show mid or late break-even behavior.

Model Performance and Fine-Tuning Effects

The analysis reveals that zero-shot foundation models are already competitive, outperforming fully trained classical methods on 19 of 30 datasets. This suggests that GPU deployment is unconditionally justified on these datasets regardless of data volume. p. 0

Where the paper says it

Across 30 benchmark datasets, we compare zero-shot and LoRA fine-tuned foundation models (Chronos, Moirai, LagLlama) against classical baselines (Naive, ETS, ARIMA, XGBoost) at six training set sizes from 2% to 100% of available data. Foundation models outperform classical methods at every evaluated training fraction on 15 of 30 datasets—GPU deployment is unconditionally justified on these regardless of data volume.

The effect of fine-tuning varies significantly depending on the series characteristics. p. 7

Where the paper says it

57) have higher raw noise but show late break-even: data starvation (H1) dominates there, masking any innovation effect. Hypothesis 3: Structural regime determines FM dominance for long series.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Automated Deployment Decision Framework: An improved system can implement a two-step deployment procedure based on training samples (ntrain) and seasonal strength (S). Step 1 resolves FM-vs-classical decision by checking if ntrain is less than 700 and S is greater than or equal to 0.05, using the rule: "If ntrain < 700 and S ≥ 0.05, use FM zero-shot. Skip fine-tuning."

  2. Pilot Experiment Triggering: For cases not resolved by the data starvation rule (ntrain < 700 and S < 0.05), the system can trigger a pilot experiment by training both FM zero-shot and XGBoost on a fraction of data to guide full deployment, as stated in Step 2: Pilot experiment.

  3. LoRA Fine-Tuning Control: The AI system should be configured to actively avoid LoRA fine-tuning when ntrain is less than 700 and S is greater than or equal to 0.05, as this regime has been shown to be where LoRA on ILI—an FM-dominant short dataset—degraded performance from MASE 2.472 to 2.766.

Abstract

Deploying a time series foundation model (FM) requires GPU infrastructure and engineering overhead, with no guarantee of improvement over classical forecasting methods. We provide the first systematic break-even analysis of when this investment pays off. Across 30 benchmark datasets, we compare zero-shot and LoRA fine-tuned FMs (Chronos-Bolt, Moirai, Lag-Llama) against classical baselines (Naive, ETS, ARIMA, XGBoost) at six training set sizes from 2% to 100% of available data. FMs outperform classical methods at every evaluated training fraction on 16 of 30 datasets. On 6 datasets, classical methods surpass zero-shot FMs with at most 5% of the training data (21--6,324 samples); on the remaining 8, break-even ranges from 24 to 22,177 samples. No classical method dominates: ARIMA, Naive, ETS and XGBoost are each best on 6--11 datasets. One robust rule requires no model training: if n train < 700 and seasonality is non-negligible, FM zero-shot is the right low-data choice -- below 20% of the history, classical methods either lose to it (4 of these 10 datasets) or cannot be fitted at all (6) -- and it stays best at full history on 6 of them; fine-tuning should be skipped. Contrary to common practice, LoRA fine-tuning rarely helps and degrades short series by up to 52%. We operationalise these findings as a two-step decision framework -- compute dataset length and seasonality strength, run a brief 5--10% pilot only if needed. Cheap dataset features do not predict the break-even category at this benchmark scale, so outside the rule a pilot remains necessary. Code and benchmark: https://github.com/nicolaisi/fm breakeven.

Sources

Related papers