IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models

arXiv:2610.16652 · cs.LG · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models".

Jane: The gist IND-Eval introduces a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, this paper is called "IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models." Basically, it introduces a public benchmark that organizes industrial time series across different domains into source-specific forecasting tasks.

Jane: That sounds like it’s trying to give us a standardized way to test these large models against real-world industrial problems, right? What's the main claim they are making here?

Tom: The main thing is that by doing this, you can expose a specialization versus generality trade-off. They show that how you evaluate a model really depends on which specific source data you are looking at.

Lu: They set up one hundred eighty-seven source-defined forecasting tasks across nine public datasets, and they emphasize finer sampling resolutions and a bigger focus on multivariate forecasting compared to older benchmarks like GIFT-Eval <ref:2610.16652#pg3>.

Meng: So, what does this mean for us in terms of actually training something? If you train a model on these tasks, are you supposed to follow the source-specific way they test it?

Jane: Exactly. The paper claims that domain-representative evaluation can guide both model selection and simulator design by showing how performance varies by source. It’s about understanding where the model is strong or weak for specific industrial settings.

Tom: And they found some interesting results when evaluating Chronos-two TiRex-two and TimesFM-three under the IND-Eval protocol using source-balanced nMASE and nCRPS following GIFT-Eval conventions <ref:2610.16652#pg1,evaluating Chronos-2, TiRex-2, and TimesFM-3>.

Jane: And what did those specific models do? What was the standout finding from that comparison?

Tom: TimesFM-three achieved the lowest source-balanced aggregate scores for both metrics, and it led seven of nine sources under each metric <ref:2610.16652#pg1,seven of nine sources under each metric>. But they immediately point out that you have to read the task taxonomy represented by their evaluation data because performance varies by source.

Lu: That finding reinforces the idea that model selection should be based on the intended use case rather than just a benchmark-wide ranking across all those metrics.

Meng: If TimesFM-three is best overall according to these aggregate scores, does that mean it's the one you should deploy right away <ref:2610.16652#pg1>? What about the caveats they mentioned?

Jane: The caveat is crucial. They are saying that because performance varies by source, aggregate scores don't tell you which model is best for every single setting. You need to look at the specific source tasks where it excels.

Tom: And that brings us into the second part of the paper, which looks at how this evaluation can guide simulator design. They investigate if a domain’s statistical structure can help improve coverage when designing a simulator using an oscillator-augmented simulation pipeline.

Jane: So, if we have scarce real industrial data, they suggest training models on simulations that are guided by the statistical structure of the target domain to fill in those gaps.

Paper summary: Lu: In a case study involving two offshore well-monitoring regimes, they found that a fifty/fifty mixture of the general and augmented simulation recipes achieved the lowest error among three training recipes for all four event-metric pairs in that specific case.

Meng: So, it sounds like tailoring the simulation pipeline to match domain statistics actually helps recover some of the performance lost when you don't have enough real data. But how much does that actually cost in terms of setup?

Tom: The paper suggests this approach reveals a trade-off between specialization and generality. It shows that you can gain better accuracy on specific events, but you might lose some overall generality across different settings.

Jane: That's the core tension they are highlighting: do you want a model that works everywhere, or one that is extremely good at one thing? This whole IND-Eval concept seems designed to help us find that sweet spot.

Lu: They also looked at a case study on severe slugging and flow instability from threeW, where their generic SARIMA-based simulation pipeline, SarSim0, poorly matched the regimes' smoothness and spectral entropy.

Meng: What were those specific statistics they used to diagnose SarSim0’s gaps? Was it just looking at the forecast error numbers?

Jane: No, they used per-window statistics to diagnose things like roughness, spectral entropy, and temporal persistence as measures of how well the simulator matched the actual physical behavior.

Tom: And when you compared that generic SarSim0 against an oscillator-augmented recipe, that augmented recipe substantially improved this measured alignment. Greater measured overlap with the target domain accompanies lower forecasting error on these specific events.

Lu: So, the paper argues that this kind of statistical guiding simulation design is consistent with what we expect from a well-designed simulator for those regimes.

Jane: It seems like they are linking model selection and simulator composition together through this idea of domain-representative evaluation. What’s the bigger picture here?

Tom: The big picture is that this evaluation method isn't just about picking the best AI model; it's about designing better data and simulations to train those models more effectively for specific industrial uses.

Meng: If we look at what they did, they are essentially showing how you can use a structured evaluation system to guide the entire pipeline from data simulation right up to the final model choice. But what’s the limitation here?

Jane: The authors themselves pointed out that this is not about broad superiority or deployment readiness, but specifically about specialization. They aren't claiming these results mean it's perfect for any general application yet.

Lu: They also noted that SarSim0 remains strongest on the native-univariate GIFT-Eval subset, which shows the cost associated with specializing the simulation design.

Tom: So, to sum up, IND-Eval shows us how domain evaluation exposes a specialization versus generality trade-off in both model selection and simulator composition. It’s a tool for designing better systems for specific industrial needs.

Conclusion: Tom: So we’ve seen how this IND-Eval benchmark lets us test these models in real industrial settings by showing a trade-off between specialization and generality across different sources and simulations.

Jane: Exactly. The authors are proposing that this isn't just about finding the best model, but it's about designing better data and better simulations to make those models actually useful for specific jobs.

Tom: They’re calling this IND-Eval, and they put a lot of focus on how you can use these domain-specific evaluations to guide both what model you pick and how you build your simulation pipeline.

Jane: So, if I'm listening right now, what does this paper actually change for someone who just listens to the show?

Tom: It means that when we look at a new AI model, we can't just judge it on one big number across everything; we have to know exactly which industrial task or source it’s actually good at.

Jane: And they show how this helps with simulations too. If you want to build a simulator for, say, offshore well monitoring, you can use the statistical structure of that domain to guide the simulation recipe, which helps it work better even when your real data is scarce.

Tom: It’s about making sure the AI isn't just generally smart across all datasets; it’s smart enough for a specific job and doesn't break when you try to deploy it in a particular environment.

Jane: So the big idea is that domain-representative evaluation gives us a way to expose that trade-off, showing us exactly where we lean toward specialization versus generality.

Tom: Right. And the authors point out that this isn't about finding some perfect, all-encompassing solution yet; it’s about giving engineers a clearer path to designing systems tailored for specific industrial needs.

Jane: It seems like the next step is figuring out how to actually use these evaluation results to bake this specialization into the AI training process itself.

Vebjørn Haugland, Pavel Gulin, David Foss Peet Cremer, Erlend Aune Simone Casolo, Andreas Solbrå Milad Mobarhan

Cognite

cs.LG

Submitted: 2026-10-12

Updated: 2026-10-12

Comments: 19 pages, 5 figures. Accepted to the Foundation Models for Temporal Systems workshop at NeurIPS 2026. IND-Eval: https://huggingface.co/datasets/cognitedata/IND-eval

Code: https://github.com/Nixtla/statsforecast

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist IND-Eval introduces a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks, revealing that domain-representative evaluation can guide

Key concepts

IND-Eval Benchmark: This is a new public test suite with 187 forecasting tasks from nine industrial datasets. It focuses on finer data sampling and more complex, multivariate forecasting than older benchmarks, allowing for source-specific performance analysis.
Specialization–Generality Trade-off: This concept describes the challenge of balancing a model's ability to perform very well on one specific industrial domain (specialization) against its ability to perform reasonably well across many different domains (generality). The study shows this trade-off influences both which AI model is best and how simulators should be built.
Oscillator-Augmented Simulation: This technique involves improving a general simulation pipeline by adding oscillator components. In testing, this augmentation helped a generic simulator better match the statistical structure of specific real-world industrial regimes, leading to lower forecasting errors for those events.

Terminology

Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes

Summary

The gist IND-Eval introduces a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks, revealing that domain-representative evaluation can guide model selection and simulator design by exposing a specialization–generality trade-off.

IND-Eval Benchmark Structure

IND-Eval comprises 187 source-defined forecasting tasks from nine public datasets, emphasizing finer sampling resolutions and a greater prevalence of multivariate forecasting compared to established benchmarks like GIFT-Eval. The benchmark includes six core sources with task structures underrepresented in established benchmarks and three industrially relevant sources shared with GIFT-Eval. A task pairs one source-specific series group with one forecast horizon and scores every target variate in each context–target window.

Model Performance Findings

When evaluating Chronos-2, TiRex-2, and TimesFM-3 under the IND-Eval protocol using source-balanced nMASE and nCRPS following GIFT-Eval conventions, TimesFM-3 achieved the lowest source-balanced aggregate scores. However, results should be interpreted within the task taxonomy represented by the evaluation data because performance varies by source. This finding reinforces that model selection should be aligned with the intended use case rather than a benchmark-wide ranking.

Simulator Composition and Domain Alignment

The study investigated whether a domain’s statistical structure can guide simulator design by improving coverage using an oscillator-augmented simulation pipeline on two offshore well-monitoring regimes <ref:2610.16652 pg3, In a case study on two offshore well-monitoring regimes, we improve a general simulator’s coverage using an oscillator-augmented simulation pipeline <ref:2610.16652#pg4>. The 50/50 mixture of the general and augmented simulation recipes achieved the lowest error among the three training recipes for all four event–metric pairs in the case study while recovering most of the lost GIFT-Eval performance. This illustrates how domain-representative evaluation can guide model selection and the composition of simulated data while revealing a trade-off between specialization and generality.

Statistical Guiding Simulation Case Study

The exploratory case study on severe slugging and flow instability from 3W showed that SarSim0, a generic SARIMA-based simulation pipeline, poorly matched these regimes’ smoothness, spectral entropy, and temporal persistence <ref:2610.16652 pg3, Simple per-window statistics diagnose SarSim0’s largest gaps with 3W: roughness, spectral entropy, and temporal persistence <ref:2610.16652#pg4>. The oscillator-augmented recipe substantially improved this measured alignment. In the exploratory comparison, greater measured overlap accompanies lower forecasting error on these events, consistent with statistic-guided simulator design.

Conclusion and Trade-off

The results illustrate how domain-representative evaluation can guide model selection and the composition of simulated data while revealing a trade-off between specialization and generality. The findings link domain-representative evaluation to model selection and simulator design, exposing a specialization–generality trade-off. The study concludes that SarSim0 remains strongest on the native-univariate GIFT-Eval subset, showing the cost of specialization.

Limitations

IND-Eval is a first release covering nine public sources and forecasting only, and we compare three released TSFMs zero-shot against naive baselines without fine-tuned or classical baselines. The simulation study is target- and score-informed with no independent design holdout. Statistical alignment is assessed using selected summary statistics, t-SNE projections, and window-level nearest-neighbor checks with frequencies expressed in cycles per sample and lags in samples. The observed target-domain gains and GIFT-Eval loss describe specialization, not broad superiority, deployment readiness, or regulatory compliance.

The paper is 2610.16652.

--- Page 9 ---

IND-Eval Benchmark Structure

IND-Eval comprises 187 source-defined forecasting tasks from nine public datasets, emphasizing finer sampling resolutions and a greater prevalence of multivariate forecasting compared to established benchmarks like GIFT-Eval. A task pairs one source-specific series group with one forecast horizon and scores every target variate in each context–target window.

Improvements for AI systems

textbf Domain-representative model selection guidance: The system can select models based on source-specific performance rather than aggregate scores, as Results should instead be interpreted within the task taxonomy represented by the evaluation data and source-specific exceptions reinforce the need for domain-representative evaluation.
textbf Simulator design guided by target statistics: The system can improve simulator coverage to match target domain characteristics, specifically by constructing an oscillator-augmented simulation pipeline when a generic simulator poorly matches these regimes’ smoothness, spectral entropy, and temporal persistence, thereby reducing forecasting error in the target domain.
textbf Trade-off management between specialization and generality: The system can balance performance across different data compositions by employing a 50/50 mixture of the general and augmented simulation recipes to achieve the lowest error among the three training recipes for all four event–metric pairs in the case study while recovering most of the lost GIFT-Eval performance.
textbf Fine-tuning strategy selection: When target data are scarce, the system can strategically choose between fine-tuning and targeted simulation, as it investigates whether a domain’s statistical structure can guide simulator design to complement sparse real industrial data.

Abstract

We introduce IND-Eval, a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks. Relative to established benchmarks such as GIFT-Eval, it emphasizes finer sampling resolutions and a greater prevalence of multivariate forecasting. Evaluating Chronos-2, TiRex-2, and TimesFM-3, we find that TimesFM-3 achieves the lowest source-balanced aggregate nMASE and nCRPS and leads seven of nine sources under each metric; source-specific exceptions reinforce the need for domain-representative evaluation for model selection. These findings also motivate improving domain alignment during pretraining, where real industrial data are often too scarce to close domain-specific gaps directly; we therefore investigate whether a domain's statistical structure can guide simulator design. In a case study on two offshore well-monitoring regimes, we improve a general simulator's coverage using an oscillator-augmented simulation pipeline. This lowers forecasting error in the target domain but reduces performance on the subset of GIFT-Eval with natively univariate targets. A 50/50 mixture of the general and augmented simulation recipes achieves the lowest error among the three training recipes for all four event-metric pairs in the case study while recovering most of the lost GIFT-Eval performance. Together, these results illustrate how domain-representative evaluation can guide model selection and the composition of simulated data while revealing a trade-off between specialization and generality.