Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting

arXiv:2610.12232 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Training on the Future".

Jane: The gist The delay-aware audit reveals that under causal, delay-aware evaluation, properly configured closed-form correctors set a bar that the audited stack clears exactly once across five datasets,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this paper now. It’s titled "Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting." This whole thing is about how we test these test-time adaptation methods when there's a delay in getting the correct label.

Jane: Exactly. The core idea is that when you update a model during forecasting, the label for that forecast doesn't come right away, and real data pipelines add even more delay to that process. This paper builds this leakage-free harness to make sure we test things correctly by setting a rule: the label of forecast origin s is only released for updates at step s plus d with d greater than or equal to H.

Lu: So they’re explicitly modeling pipeline latency, where d equals H is the minimum causal delay, and when d is greater than H, you’re modeling actual latency in the pipeline. They even set d equals one to reproduce that leaky legacy protocol we all see in some older systems <ref:2610.12232#pg2>.

Meng: That sounds like a very precise way to control the experiment environment so you can actually measure adaptation effects instead of just seeing noise from delayed information. But what does this audit actually find when they run it across five different benchmarks?

Tom: Well, the paper audits four recent test-time adaptation methods—TAFAS, COSA, PETSA and DynaTTA—on five datasets: ETTm1, ETTh2, Weather, Electricity and Traffic. The main point is that under this causal delay setting they tested on these five benchmarks, the closed-form correctors set a bar that the audited stack clears exactly once across all those datasets.

Jane: That’s a strong statement. It suggests that simply using an adaptation method isn't enough; there’s a specific way you have to structure the evaluation to see if it actually works under real-world constraints.

Lu: And they show that this evaluation practice, not just the methods themselves, needs serious recalibration in how we test these techniques for time-series forecasting.

Meng: The paper then introduces two closed-form reference adapters as baselines for comparison: a bank of recursive least squares filters combined by a per-coordinate median and an ELF-style linear corrector. The RLS-Bank has no tunable hyperparameters at deployment, and it costs fifty six microseconds per step on the seven-channel streams.

Tom: That cost is pretty low compared to some of the other methods they’re testing, which is important for practical use. They also say that under causal delayed labels, the RLS-Bank genuinely adapts on ETTm1 while beating three out of four of those methods at a fraction of their cost.

Jane: So we see a clear advantage in terms of both performance and efficiency when you stick to this causal evaluation framework. But what about the other datasets?

Title and authors: Lu: On the other datasets, they found that the largest statistically significant improvement any published method achieves over its own frozen checkpoint is only half a percent.

Meng: And for all four of those other datasets, at least one published method is actually significantly worse than the frozen model when you use the minimum causal delay setting. That’s a pretty sobering result for some of these adaptations.

Tom: They also found that using leaky next-step updates inflates the apparent gains of simple adapters by up to one hundred ten percent, which means those easy wins aren't as good as they look when you factor in the delay issue.

Jane: And they pointed out that just training on the backbone recipe itself can move online error by a factor of twenty-five, which is more than any adaptation effect they measured. That suggests focusing on the training setup itself has a bigger impact than tweaking the adapter.

Lu: The paper also looked at robustness and failure modes, specifically with delayed labels. The RLS-Bank’s median selection works well for absorbing numerically exploding experts on the thirty-two- and eight-sixty-channel streams because it absorbs them, staying within eight percent of the best single lambda.

Meng: But they found a problem too: the feedback-driven expert selection fails under delayed labels because it suffers from delayed-feedback pathology, where a hard selector keeps spitting out an expert’s predictions for up to d steps after that expert has actually started to fail.

Tom: This bank turns harmful on ETTh2 at extended delays, specifically when d equals two times H, seeing increases of zero point one six to zero point twenty-six across the three checkpoint seeds. That shows where the method breaks down under more realistic pipeline latency scenarios.

Jane: So the audit is really highlighting that evaluation practice needs a serious recalibration because just looking at adaptation metrics isn't enough; you have to account for how data arrives and when it arrives. What does this mean for how we should be testing these models moving forward?

Lu: The paper distills the audit into five recommendations for evaluating test-time adaptation in forecasting. These include never updating on unmatured windows, reporting d equals H as the primary setting and at least one d greater than H, including persistence, seasonal-naive and the frozen model in the same table as those adapted models.

Meng: They also recommend reporting adaptation cost right next to accuracy, and reporting the backbone training recipe because it moves online error by up to a factor of twenty-five. And critically, they say including closed-form correctors as mandatory baselines is essential for setting a proper performance bar.

Title and authors: Tom: So the big message from "Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting" is that evaluation practice needs a complete recalibration. It’s not just about which adaptation technique performs best; it’s about how you measure those techniques under realistic delay conditions.

Jane: It really emphasizes that we have to stop assuming ideal data flow and start building in the uncertainty of real pipelines when we're testing these kinds of systems. This audit makes it clear that simply chasing a high accuracy score on one dataset isn't enough if you haven't tested how that method behaves when the label is delayed by a reasonable amount.

Lu: And for future work, they point out recipe sensitivity as a major system-level finding. They showed that the same architecture, like PatchTST on ETTm1, can score six point eight zero frozen under one training recipe and only one point three seven under another authors' recipe, which is a factor of five difference.

Meng: And they also found this ranking inversion—adaptation actually changes which backbone you should deploy. That’s a big thing for engineers because it means the choice of model structure becomes dependent on the adaptation strategy you choose.

Tom: So we have some concrete numbers here, like the RLS-Bank costing fifty six microseconds per step, and system-level findings showing that recipe sensitivity can lead to a factor of twenty-five difference in online error. It’s all about making sure the evaluation reflects what happens when you actually deploy these things.

Jane: It’s a lot of detail, but the practical implication is that practitioners need to be much more rigorous about their testing protocols for any test-time adaptation method they use in forecasting. We can't just rely on simple metrics anymore if we want to get reliable results that hold up in the real world.

Lu: To wrap up this whole discussion on "Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting," remember that closed-form correctors are setting a bar that most adapted stacks can't clear under causal, delay-aware evaluation. It’s not about throwing away adaptation; it’s about making sure your evaluation practice is sound.

Meng: I think the key thing to remember is the cost analysis—the RLS-Bank being two to four orders of magnitude cheaper than gradient methods on those streams. That’s a huge practical win for deployment.

Tom: Yeah, and we need to keep looking at these system-level findings, like how adaptation can actually change which backbone you deploy. It's not just about getting the highest accuracy on paper; it’s about deploying something that makes sense in practice.

Jane: So let's take this idea—the delay-aware evaluation protocol—and start thinking about how we can integrate it into our standard testing procedures for these kinds of models. That’s where the real progress is coming from right now.

The paper's summary: Tom: So, we’ve seen that this paper is basically a deep dive audit of how we test these test-time adaptation methods in time-series forecasting, focusing specifically on those delays in getting the correct label.

Jane: Right, and they set up this very specific setup where they force you to model real pipeline latency by making sure the label release happens only after a certain delay.

Tom: What’s really interesting is what they found when they ran these methods across five different datasets—they showed that the closed-form correctors, like the RLS-Bank, are setting a performance standard that most of those adapted methods just can't reach.

Jane: And it’s not just about the accuracy numbers; it’s about how these adaptation techniques actually behave when they have to deal with these real-world delays.

Tom: They found that simple adapters can look way better than they are, sometimes inflating their gains by up to one hundred ten percent because you're using future targets that haven't happened yet.

Jane: That means we have to be much more careful about trusting those adaptation metrics if we aren't accounting for the delay structure of the data flow.

Tom: And they also showed that just training on the backbone recipe itself can move online error by a factor of twenty-five, which is bigger than most of the adaptation effects they measured.

Jane: So it suggests that focusing purely on tweaking those adaptation layers might not be as impactful as fixing how you’re actually training the model from the start.

Tom: On a system level, they found some pretty big things, like recipe sensitivity where different training setups gave vastly different scores for the exact same architecture on those time-series streams.

Jane: And another big thing was that adaptation itself could actually decide which backbone you should even deploy for the best results.

Tom: It’s a lot of detail, but it really drives home that evaluation practice needs a complete rethink if we want to get reliable results in this area.

Jane: We’re going to talk more about those specific five recommendations they give for how we should be setting up our tests moving forward, so let's break down what those look like.

The paper's improvements: Tom: So, we’ve just heard that this paper isn't just pointing out problems; they’ve actually laid out a whole set of fixes for how we should be testing these test-time adaptation methods in forecasting.

Jane: Yes, they distill all their findings into five concrete recommendations for researchers on what to do next when they are trying to evaluate these models.

Tom: The first big thing is that you should never update a model on windows that haven't matured yet; that’s a crucial rule for avoiding bad results.

Jane: And they stress reporting d equals H as your main setting, and then testing at least one delay greater than H, like including persistence or seasonal-naive models.

Tom: They also want you to put the frozen model right alongside your adapted models in the same table so you can compare them directly.

Jane: Plus, they say including those closed-form correctors we talked about—the RLS-Bank and the ELF-style corrector—as mandatory baselines for comparison is essential.

Tom: That’s a strong call because it sets a performance bar that many of the gradient methods they audited just don't manage to clear across those five different datasets.

Jane: They also want you to report the adaptation cost right next to the accuracy score so you can see how much computational effort you’re actually putting in.

Tom: And they want you to report that backbone training recipe, because it moves the online error by a factor of twenty-five, which is more than any of their adaptation effects they measured.

Jane: So it pushes everyone to look beyond just the final accuracy score and consider the entire pipeline—the training setup and the latency structure.

Tom: It really changes how we think about testing these things; it's not just about finding an adaptation method that works well, but making sure your whole evaluation practice is sound under real-world constraints.

Jane: Exactly, because if you don’t follow these five steps, you risk overestimating how good your model is performing in a practical setting.

Tom: The implication here is that we need to start standardizing our testing protocols so that everyone is playing by the same rules when evaluating test-time adaptation techniques for time-series forecasting.

Conclusion: Tom: So, to wrap up, this paper, "Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting," shows that we need to completely rethink how we evaluate these models when data labels have a delay.

Jane: Exactly. It boils down to this: simply checking accuracy isn't enough if you aren't modeling the real delay structure of the pipeline during testing.

Tom: The main implication is that practitioners have to adopt these new five evaluation steps—like using d equals H as your primary setting and including those closed-form correctors as mandatory baselines.

Jane: It means we can’t just pick a method that scores high on one dataset and assume it will work everywhere without testing it under causal delay conditions.

Tom: And they found some pretty big system-level findings, like adaptation actually changing which backbone you should deploy, which is a huge practical headache for engineers.

Jane: It really emphasizes that the choice of model structure becomes dependent on the adaptation strategy you choose.

Lu: From a theoretical standpoint, this audit gives us a much clearer picture of the causal dependencies in these time-series forecasting systems, opening up new avenues for designing more robust architectures.

Meng: For me, it means we have to stop optimizing just for the best score and start optimizing for stability under real operational constraints where delays are guaranteed.

Lalam: And this kind of rigorous evaluation practice helps improve culture because it forces us to build systems that are inherently more resilient, which is a vital step toward creating trustworthy AI applications.

Tom: Yeah, so it’s not just about the algorithms anymore; it’s about the entire testing philosophy.

Jane: It's a call for much more rigorous evaluation protocols in forecasting research.

Tom: That's all we have time for today on this paper, "Training on the Future: A Delay-Aware Audit of Test-Time Adaptation for Time-Series Forecasting." We’ll be back next week to talk about how other papers are tackling these kinds of real-world deployment issues.

Mohamed Readh Fentazi, Mazene Ameur, Adlen Ksentini

EURECOM

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 34 pages, 9 figures. Code and cached results: https://github.com/Xodios/TRAINING-ON-THE-FUTURE-A-DELAY-AWARE-AUDIT-OF-TEST-TIME-ADAPTATION-FOR-TIME-SERIES-FORECASTING

Code: https://github.com/Xodios/TRAINING-ON-THE-FUTURE-A-DELAY-AWARE-AUDIT-OF-TEST-TIME-ADAPTATION-FOR-TIME-SERIES-FORECASTING

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist The delay-aware audit reveals that under causal, delay-aware evaluation, properly configured closed-form correctors set a bar that the audited stack clears exactly once across five datasets,

Key concepts

Delay-Aware Evaluation: This is a specific way to test time-series models where the evaluation considers a 'delay' (d) between when a forecast origin's label is released and when an update occurs. This setting tests how robust an adaptation method is to different levels of latency or lag in receiving feedback.
Closed-Form Correctors: These are specific, pre-defined mathematical formulas used as baseline methods for adapting models. The paper introduces two: an RLS-Bank and an ELF-style linear corrector. These act as simple, fixed reference points against which the performance of more complex adaptation techniques is measured.
Leakage-Free Harness: This is a controlled testing setup designed to prevent data leakage during the evaluation of test-time adaptation methods. It ensures that the label information for forecast origins is only released at a specific, delayed step, mimicking real-world scenarios where feedback arrives later than expected.
Recipe Sensitivity: This finding shows that even with the same model architecture and stream, different training recipes can lead to vastly different performance outcomes. The same model trained with one recipe might score significantly better or worse than the same model trained with another recipe.

Terminology

Adaptation — 1× in this paper · explained in 1 episode

Summary

The gist The delay-aware audit reveals that under causal, delay-aware evaluation, properly configured closed-form correctors set a bar that the audited stack clears exactly once across five datasets, and that evaluation practice, not only the methods, needs recalibration.

How it works

The paper introduces a leakage-free harness where the label of forecast origin s is released for adapter updates only at step s + d with d ≥ H. This setting d = H is the minimum causal delay, while d > H models pipeline latency, and d = 1 reproduces the leaky legacy protocol. The evaluation protocol explicitly makes this delay a variable by enforcing it inside the released code of four recent TTA methods (TAFAS, COSA, PETSA and DynaTTA).

Key Findings on Performance

The audit shows that under causal updates, the RLS-Bank genuinely adapts on ETTm1 while beating three of the four at a fraction of their cost. On other datasets, the largest statistically significant improvement any published method achieves over its own frozen checkpoint is half a percent, and on all four of the other datasets at least one published method is significantly worse than the frozen model at the minimum causal delay. Furthermore, leaky next-step updates inflate the apparent gains of simple adapters by up to 110%, and the backbone training recipe alone moves frozen online error by up to a factor of 25, more than any adaptation effect we measure.

Closed-Form Correctors as Baselines

The paper introduces two closed-form reference adapters: a bank of recursive least squares (RLS) filters combined by a per-coordinate median, and an ELF-style linear corrector. The RLS-Bank has no tunable hyperparameters at deployment and costs 56 microseconds per step on one CPU core for a 7-channel, 24-horizon problem. The ELF-style corrector is restricted to streams with C ≤ 21 because its per-step cost is O(CHL 2).

Robustness and Failure Modes

The RLS-Bank's median selection mechanism successfully absorbs numerically exploding experts on the 321- and 862-channel streams, as the median absorbs them, staying within 8% of the best single λ (Appendix B). Conversely, the feedback-driven expert selection fails under delayed labels because it suffers from delayed-feedback pathology where a hard selector keeps emitting an expert’s predictions for up to d steps after that expert has started to fail. The bank turns harmful on ETTh2 at extended delay, significantly so at d = 2H (+0.16 to +0.26 across the three checkpoint seeds) and at d = 4H (+0.33 to +0.44).

Evaluation Practice Recommendations

The paper distills the audit into five recommendations for evaluating TTA in forecasting. These include never updating on unmatured windows, reporting d = H as the primary setting and at least one d > H, including persistence, seasonal-naive and the frozen model in the same table as the adapted models, including closed-form correctors as mandatory baselines, reporting adaptation cost next to accuracy, and reporting the backbone training recipe which moves online error by up to a factor of 25. The conclusion is that evaluation practice needs recalibration.

Cost Analysis

The RLS-Bank updates 4,032 closed-form weights in 0.056 ms per stream step on one CPU core on the 7-channel ETT streams, with no tunable hyperparameters; its cost scales as O(CHF 2) and averages 2.9 ms per step over all ten cells including the 862-channel Traffic stream. This makes the bank two to four orders of magnitude cheaper than every gradient method it is compared with on those streams. The ELF-style corrector costs about 7 ms per step on the ETT-class streams because it regresses on the full lookback window (O(CHL 2) with L=96), which is also why we restrict it to C ≤ 21. The bank's remaining advantages are cost (roughly two orders of magnitude, Section 5.3) and applicability to the wide streams where the ELF-style corrector is impractical and the bank posts its clearest wins.

System-Level Findings

Two findings at the system level are larger than any method-level effect in this paper. The first is recipe sensitivity, where the same PatchTST architecture on the same stream, split and scaling scores 6.80 frozen under our training recipe and 1.37 under the authors’ recipe on ETTh2 (a factor of 5), and 86.1 against 4.76 on Electricity (a factor of 25); on ETTm1 the sign flips, with our recipe frozen at 1.14 against the authors’ 1.52. The second is a ranking inversion, where adaptation changes which backbone one should deploy. These findings suggest that adaptation changes which backbone one should deploy.

Reproducibility

All internal and external runs are cached and keyed by configuration hash, with the exact run manifest shipped in the release; the pipeline is deterministic given the backbone and was reproduced across three machines to 10-4. We release the harness notebook, the integration kit (the patcher with its full change log and the runner with alignment asserts), the stream specifications and all cached results. The COSA reference repository is CC BY-NC-SA 4.0; we distribute patches and integration scripts, not modified copies of the repository. All code was read and tested by the authors; the internal pipeline was reproduced across three machines to within 10-4. The paper's appendices provide logs, pseudocode, and bootstrap details for reproducibility. All code was read and tested by the authors; the internal pipeline was reproduced across three machines to within 10-4. The COSA reference repository is CC BY-NC-SA 4.0; we distribute patches and integration scripts, not modified copies of the repository. All code was read and tested by the authors; the internal pipeline was reproduced across three machines to within 10-4. The COSA reference repository is CC BY-NC-SA 4.0; we distribute patches and integration scripts, not modified copies of the repository. All code was read and tested by the authors; the internal pipeline was reproduced across three machines to within 10-4. The COSA reference repository is CC BY-NC-SA 4.0; we distribute patches and integration scripts, not modified copies of the repository. All code was read and tested by the authors; the internal pipeline was reproduced across three machines to within 10-4. The COSA reference repository is CC BY-NC-SA 4.0; we distribute patches and integration scripts, not modified copies of the repository. All code was read and tested by the authors; the internal pipeline was reproduced across three machines to within 10-4.

Improvements for AI systems

Bold header: Leakage-free, delay-parameterized evaluation protocol. This enables researchers to rigorously test TTA methods by enforcing a rule that the label of forecast origin s is released for updates only at step s + d with d ≥ H, explicitly modeling real pipeline delays.
Bold header: RLS-Bank closed-form adapter with no tunable hyperparameters. This adapter achieves 56 microseconds per step on the 7-channel streams and is robust because the median absorbs numerically exploding experts, providing a reliable baseline for comparison.
Bold header: Identifying recipe sensitivity in backbone training. The paper demonstrates that the backbone training recipe moves frozen online error by up to a factor of 25, more than any adaptation effect we measure, necessitating reporting the exact recipe used to ensure fair evaluation of TTA methods.
Bold header: Causal delay-aware evaluation practice for TTA. The finding leaky updates inflate the apparent gains of simple adapters by up to 110% forces practitioners to evaluate methods using a causal protocol, ensuring that adaptation effects are not overestimated by consuming future targets that have not yet been observed.
Bold header: Mandatory closed-form baselines for TTA evaluation. The audit recommends including closed-form correctors as mandatory baselines, specifically the RLS-Bank and the ELF-style corrector, to establish a performance bar that audited gradient methods often fail to meet across five datasets.