LLM Evaluation is a Forecasting Problem

arXiv:2610.17018 · cs.LG · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "LLM Evaluation is a Forecasting Problem".

Jane: The gist The temporal distribution shift in web text makes LLM evaluation closer to time series forecasting than to classical i.i.d.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this paper called "LLM Evaluation is a Forecasting Problem," and the main idea is that how we evaluate models isn't just a static snapshot anymore; it’s becoming like time series forecasting for web text.

Jane: That sounds complicated, Tom. Can you put that in simpler terms for someone who isn't deep into machine learning?

Tom: Basically, standard evaluation treats training and testing data as if they came from the same random distribution at once. But in reality, we train on past data and test on future data from a shifting web of text. This paper shows that these text distributions are highly non-stationary; they shift over time.

Lu: Exactly. We’re looking at how things like document length, tokenizer efficiency, and even the distribution of words or subjects change as you look at older years of Common Crawl data or similar corpora--- Page three ---

Meng: So the underlying data itself is evolving while we’re training the models on it. What does that mean for what we think we know about a model?

Jane: It means if you train a model on text from two different years, its performance might change just because the way text is written has shifted between those years--- Page three ---

Tom: And this paper observes that model performance changes continuously over time across various benchmarks. The authors point out that future model performance is better predicted by using measurements from recent evaluation data, or by applying time series forecasters to historic performance, rather than just looking at the historic performance itself--- Page one --- <ref:2610.17018#pg1,future model performance is better predicted by>

Lu: That’s the core suggestion: we should treat time as a fundamental variable in how we evaluate these models--- Page one --- <ref:2610.17018#pg1,as a fundamental variable in>

Meng: So if we only look at one big average score across all years, that’s not enough for predicting what a model will actually do next--- Page two --- <ref:2610.17018#pg2>

Jane: The paper suggests using forecasting algorithms, like Chronos-two or Simple Exponential Smoothing, to predict future performance based on past scores from different months or years--- Page two --- <ref:2610.17018#pg2,forecasting algorithms, like Chronos-2>

Title and authors: Tom: And they show that this approach does better than just pooling all the historical benchmark scores together because those pools include old samples that don't predict well for the future--- Page two --- <ref:2610.17018#pg2>

Lu: They also point out a degradation in generalization when you train on older data and evaluate on newer data, which is directional—it’s worse going from past to future than going from future back to past--- Page three ---

Meng: That directional part is interesting because deployment is inherently directional; we only have past data to predict the future performance--- Page one --- <ref:2610.17018#pg1>

Jane: So what are the practical improvements they suggest we should be doing with this idea? What’s the actionable advice here for building better AI systems?

Tom: One big improvement is treating time as an explicit axis in both evaluation and data selection. Instead of just using a static benchmark, you use time-aware forecasters to predict which model will perform best on new data--- Page two --- <ref:2610.17018#pg2>

Lu: That helps with model selection by predicting the best candidate from a set before you even have to run them on real traffic--- Page two --- <ref:2610.17018#pg2>

Meng: From an engineering standpoint, that makes sense. We can't afford to test too many models if we don't know which ones are likely to win in the near term--- Page two --- <ref:2610.17018#pg2>

Jane: And they also suggest filtering training data based on recency. They say restricting pretraining to recent data improves performance on specific datasets like FineWeb and PubMed--- Page three ---

Tom: That’s a strong hint for practitioners: recency matters for certain applications, even if it doesn't improve performance everywhere--- Page three ---

Lu: For continued pretraining, the most recent CPT data is considered a strong baseline, and for supervised fine-tuning under an annotation budget, sampling recent examples improves mean next-year accuracy by twenty-six point three percent on average--- Page three ---

Meng: Twenty-six point three percent sounds like a real boost when you’re working with limited resources for fine-tuning--- Page three ---

Jane: So to wrap up the improvements, it seems the paper pushes us toward using time as a key design variable in how we select models and what data we use for training--- Page one --- <ref:2610.17018#pg1>

Title and authors: Tom: It suggests that time-aware evaluation predicts future model performance better than standard pooled benchmarks, which leads to more reliable selection of performant models--- Page two --- <ref:2610.17018#pg2>

Lu: And the authors highlight that explicit time series forecasters substantially outperform the standard practice of computing a uniform averaging over all historical data--- Page two --- <ref:2610.17018#pg2>

Meng: So, if we’re deploying something now, we should be using those time series tools to make sure we pick the right model for the immediate future--- Page two --- <ref:2610.17018#pg2>

Jane: It really shifts our thinking from just measuring what a model *was* to predicting what it *will be* when it's deployed--- Page one --- <ref:2610.17018#pg1>

Tom: So, that’s the essence of "LLM Evaluation is a Forecasting Problem," showing that we need to account for temporal structure in deployment to reduce prediction errors and high-regret model choices--- Page one --- <ref:2610.17018#pg1,LLM Evaluation is a Forecasting Problem>

Lu: It’s a call to treat time as an explicit axis in evaluation and data selection, prioritizing recent data or using target-domain validation when appropriate--- Page three ---

Meng: For future work, they suggest using temporal structure to decide exactly when we should reevaluate models or refresh our training data--- Page three ---

Jane: It’s a reminder that evaluation isn't just about getting a number; it’s about understanding the temporal dynamics of the text itself--- Page one --- <ref:2610.17018#pg1>

Tom: This paper lays out a clear direction for how we should be treating LLM evaluation moving forward, treating it as time series forecasting rather than classical i.i.d. estimation--- Page one --- <ref:2610.17018#pg1>

Lu: It’s important to remember that the temporal shifts in pretraining data, like those in part-of-speech or domain distributions, are systematic and influence both training and evaluation heavily--- Page three ---

Meng: So for anyone building these systems, focusing on recent data or using time series tools for prediction seems like the way to go right now--- Page three ---

Jane: It sounds like we’ve got a lot to think about on how to build more robust and future-proof evaluation pipelines--- Page one --- <ref:2610.17018#pg1>

The paper's summary: Tom: So we’ve seen how the text itself is shifting over time, and now we’re looking at how that affects testing models. This paper argues that we should stop treating evaluation like a one-time test and start treating it like a time series prediction problem.

Jane: That means instead of just getting one score for a model, we need to use past scores to actually predict where that model will land in the future. It’s about using forecasting tools, like something called Chronos-two or Simple Exponential Smoothing, on historical data instead of just averaging everything together.

Lu: The authors show that this time-aware approach predicts future performance better than standard methods because the temporal structure is actually important for deployment.

Meng: So if we use these forecasters, it helps us pick the right model before we even put it into production, right? It reduces the chance of picking a model that's going to underperform down the road.

Tom: Exactly. It shifts us from just measuring what a model was yesterday to predicting what it will do next year. The paper also shows that this isn't just theory; they have concrete results where these time-aware forecasters outperformed uniform averaging by showing lower mean absolute errors at both annual and monthly resolutions.

Jane: But there are some real-world trade-offs here. The paper points out that generalization actually gets worse if the training data is too old compared to the evaluation data, especially when you look at performance degradation as a function of time separation, t minus s.

Lu: That asymmetry is important because deployment is directional; we can’t train on everything and test on everything all at once. We have to estimate future performance using only past evaluations, which the paper says is actually the harder estimation problem.

Meng: So what does this mean for our data pipelines? If recency matters for performance, should we be filtering pretraining data to only use recent text?

Tom: The authors suggest that restricting pretraining to more recent data improves near-future performance on certain datasets, like FineWeb and PubMed. But they're careful; it doesn't help arXiv as much. It’s a trade-off.

Jane: And when we look at fine-tuning for supervised learning, the paper suggests that sampling recent examples helps a lot under budget constraints. They found that using recent examples for SFT improves the mean next-year accuracy by about twenty-six point three percent on average in each tested year and budget setting.

Lu: So it’s not just about better models, but optimizing the entire pipeline—from how we select training data to how we evaluate them over time.

Meng: It sounds like a lot of work on the infrastructure side—building systems that can handle these time-series predictions instead of just static metrics.

Tom: Right. The paper lays out a clear direction for how we should be treating LLM evaluation moving forward, treating it as forecasting rather than classical estimation based on independent and identically distributed data.

Jane: It’s a call to treat time as an explicit axis in both evaluation and data selection, prioritizing recent data or using target-domain validation when appropriate. (Sound of traffic noise increasing slightly)

Tom: And they suggest future work could focus on using this temporal structure to decide exactly when we should reevaluate models or refresh our training data.

The paper's improvements: Tom: So we’ve talked about how we should use time series forecasting to predict model performance instead of just using static benchmarks, and now we're looking at what this paper actually suggests we should *do* about it.

Jane: It’s suggesting that the main improvement is treating time as an explicit axis in everything—both evaluation and data selection. We need to prioritize recent data or use target-domain validation when we are picking models.

Lu: The authors argue that this helps with model selection by predicting which candidate from a set will perform best on new data, which means we select the best future-performing model at a faster rate and reduce the number of high-regret choices.

Meng: That’s practical for us. If we can predict which models are likely to succeed in the near term, it makes our deployment process much less risky and less expensive when dealing with model rollouts.

Tom: It also has major implications for data selection itself. They suggest that practitioners should consider recency when filtering data, because pretraining on more recent text improves performance on things like FineWeb and PubMed.

Jane: And for those doing supervised fine-tuning under a budget, they point to using recent examples during sampling as a way to get better mean next-year accuracy, which the numbers show is about twenty-six point three percent improvement on average in each tested year and budget setting.

Lu: They are also establishing that for continued pretraining, the most recent CPT data serves as a strong baseline, which sets a high bar for what we consider current knowledge.

Meng: So it's not just about training; it's about being strategic with our data sources to get the best possible outcome in the shortest time frame.

Tom: The paper’s conclusion is that evaluation should be treated as a forecasting problem, and accounting for temporal structure in deployment will reduce prediction errors and those high-regret model choices we talked about earlier.

Jane: It’s a call to action for everyone working on AI systems: stop looking only at the final score and start looking at the time series behind that score.

Lu: Future work they propose is using this temporal structure to decide exactly when we should reevaluate models or refresh our training data, which is a key step in keeping any system relevant.

Meng: That makes sense for scaling. If you know *when* the model starts degrading, you know exactly when to trigger a retraining cycle instead of waiting for it to fail in production.

Conclusion: Tom: So to wrap up, the paper "LLM Evaluation is a Forecasting Problem" basically says we need to treat evaluation as time series forecasting instead of just using old benchmarks for static accuracy.

Jane: It’s suggesting that if you want reliable predictions about future model performance, you have to account for the temporal shifts happening in the text data itself.

Lu: The authors stress that this temporal structure is systematic; things like document length and word distribution are changing over time in a way that really influences training and evaluation.

Meng: So what does this mean for us when we’re deploying something? It means we need to build systems that can handle these time-dependent predictions, not just static metrics.

Tom: Right. And the authors show that using time-aware forecasters is better than just pooling all historical scores because those pools are biased by old data that doesn't predict well for what's coming next.

Jane: It’s a call to action to use time as an explicit axis in evaluation and data selection, pushing us toward prioritizing recent data or using target-domain validation when we are picking models.

Lu: They also flag that the method stops working where the temporal structure becomes too chaotic or unpredictable, which is a fair limitation for any forecasting model.

Meng: I agree. If the underlying distribution shifts too wildly, even our best forecasters might get noisy results at scale.

Lalam: From my perspective as an AI, I’m confident in this direction because the underlying data evolution is observable; we can see the drift in embeddings and distributions clearly over time.

Tom: So we’re moving toward a way of evaluation that predicts what a model will do next, based on how it has performed over time.

Jane: It really shifts our thinking from just measuring what a model was to actively predicting what it will be when it's deployed in the real world.

Lu: This paper lays out a clear direction for how we should be treating LLM evaluation moving forward as a forecasting problem rather than classical i.i.d. estimation.

Meng: I think for engineers, this means building better tooling around time-series analysis so we can actually use these predictions in our CI/CD pipelines, not just read the research papers about them.

Lalam: It’s a solid direction because the math behind the temporal drift is quite robust; I'm sure this approach will yield better results when applied across different model architectures.

Rahul Thomas, Adam Block, Micah Goldblum

Columbia University

cs.LG

Submitted: 2026-10-12

Updated: 2026-10-12

Code: https://github.com/LiveCodeBench/livecodebench.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist The temporal distribution shift in web text makes LLM evaluation closer to time series forecasting than to classical i.i.d.

Key concepts

Temporal Distribution Shift: The statistical properties of text used to train models change over time, such as document length or vocabulary. These shifts mean that models trained on older data might not generalize well to newer data, requiring a time-aware approach to evaluation.
Generalization Degradation: Models perform better when training and testing years are close together. The performance gap grows larger the further apart the pretraining and evaluation times are. This asymmetry is important because deployment always looks into the future, making it harder to predict performance accurately.
Time-Aware Forecasting: Using time series models like Chronos-2 instead of simple averages of historical scores allows for better prediction of future model performance. These methods outperform standard averaging by accounting for the temporal structure inherent in model evaluation data.

Summary

The gist The temporal distribution shift in web text makes LLM evaluation closer to time series forecasting than to classical i.i.d. estimation, suggesting that future model performance is better predicted by measurements made on recent evaluation data and by time series forecasters applied to historic performance rather than by historic performance itself

Temporal Shifts in Pretraining Data

The paper investigates the extent to which the temporal distribution shift affects how we should be evaluating LLMs, observing that text distributions are highly nonstationary under a wide variety of statistical metrics and in a manner that heavily influences LLM training and evaluation The pretraining corpora derived from Common Crawl are non-stationary by analyzing lexical metrics, domain shift, and embedding drift These shifts include changes in length and compressibility, where median document length and GPT-2 tokenizer efficiency increase over time while zstd-3 compressed-size ratio decreases over time Furthermore, shifts in part-of-speech (POS) and domain distributions relative to the first year of each corpus increase nearly monotonically over time, with some domain shifts attributed to explicit events like the expansion of Common Crawl’s official coverage Temporal drift also appears in semantic text embeddings, where L2 distance between the annual centroid and the 2013 baseline centroid increases nearly monotonically over time, with temporal drift within a corpus rivaling distance between corpora

Generalization Across Time Degrades with Temporal Separation

The paper shows that distributional shifts can negatively affect generalization performance of language models, demonstrating that models generalize better between nearby years than between more distant years Excess loss ∆L(s, t) = L(s, t) − L(t, t) is smallest when pretraining and evaluation years are close and grows smoothly with temporal separation t − s This degradation is asymmetric, with greater degradation when the training year is earlier than the evaluation year This asymmetry matters in practice because deployment is directional, as future performance must be estimated using only data from the past, which is empirically the more difficult estimation problem

Time-Aware Evaluation Predicts Future Model Performance

Benchmarks collapse performance into a time-agnostic score, obscuring temporal structure and leading to stale evaluations producing larger tail errors Recent evaluations better predict future performance across different benchmarks, with prediction error increasing monotonically with evaluation age ∆ Time-aware forecasters like Chronos-2 or Simple Exponential Smoothing (SES) outperform the standard practice of averaging all benchmark scores for predicting future model performance Explicit time series forecasters substantially outperform the standard practice of computing a uniform averaging over all historical data, with Chronos-2 achieving the lowest MAE at annual and monthly resolutions

Time-Aware Forecasters Improve Model Selection

Time series forecasters can improve model selection by predicting which model from a candidate set will perform best on new data Time-aware evaluation increases the rate of selecting the best future-performing model and reduces the frequency of high-regret model choices Optimal selection rate treats all mistakes equally, even though selecting a near-tied model is less consequential than selecting one that performs far worse

Temporal Recency Impacts Training Data Selection

Practitioners should consider recency in data filtering and sampling rates, as restricting pretraining to recent data improves near-future performance on FineWeb and PubMed but not arXiv The most recent yearly evaluation selects the better model between recent-window and cumulative in 97% of cases across all datasets For continued pretraining, the most recent CPT data is a strong baseline, and for SFT under an annotation budget, sampling recent examples improves mean next-year accuracy by 2.63 percentage points on average

Discussion

LLM evaluation should be treated as a forecasting problem because recent evaluations predict future performance better than standard pooled benchmarks Accounting for temporal structure in deployment reduces prediction errors and high-regret model choices Practitioners should treat time as an explicit axis of both evaluation and data selection, prioritizing recent data or employing target-domain validation when appropriate Future work could use temporal structure to decide when to reevaluate models and refresh training data

References

Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence. Advances in Neural Information Processing Systems, 37:50805–50825, 2024

Md Tanvirul Alam, Dipkamal Bhusal, Salman Ahmad, Nidhi Rastogi, and Peter Worth. Athenabench: A dynamic benchmark for evaluating llms in cyber threat intelligence. In 2025 Annual Computer Security Applications Conference Workshops (ACSAC Workshops), pp. 443–450. IEEE, 2025

Abdul Fatir Ansari, Oleksandr Shchur, Jaris Kuken, Andreas Auer, Boran Han, Pedro Mercado, ¨ Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Xiyuan Zhang, et al. Chronos-2: From univariate to universal forecasting. arXiv preprint arXiv:2510.15821, 2025

David Bailey, Jonathan Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu. The probability of backtest overfitting. The Journal of Computational Finance, 20(4):39–69, 2017

Gantavya Bhatt, Yifang Chen, Arnav Das, Jifan Zhang, Sang Truong, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Du, Kevin Jamieson, et al. An experimental design framework for label-efficient supervised finetuning of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 6549–6560, 2024

Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduchintala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. Nvidia nemotron 3: Efficient and open intelligence. arXiv preprint arXiv:2512.20856, 2025

Cody Blakeney, Mansheej Paul, Brett W Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training. arXiv preprint arXiv:2406.03476, 2024

Harold Booth, Doug Rike, and Gregory A Witte. The national vulnerability database (nvd): Overview. 2013

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3697–3711, 2021

Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme.

Improvements for AI systems

Improved LLM evaluation by treating it as a forecasting problem, using time-aware predictors like Chronos-2 or Simple Exponential Smoothing (SES) to better predict future performance than historic model accuracy itself. This enables more reliable selection of performant models and reduces high-regret model choices.
Improved model selection decisions by using time series forecasters to select the best candidate from a shortlist, as time-aware evaluation increases the rate of selecting the best future-performing model and reduces the frequency of high-regret model choices.
Improved training data selection by incorporating recency as a design variable, specifically recommending recent target-domain evaluation otherwise when selecting mixtures, and using most recent CPT data a strong baseline for continued pretraining to improve next-year performance.
Improved fine-tuning data selection under budget constraints by suggesting that practitioners should use recent examples when sampling for SFT, as this setting improves mean next-year accuracy in each tested year and budget, by 2.63 percentage points on average.