Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: We're looking at the paper: "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes".

Warren: Here's the gist of the paper. This paper examines the use of recurrent neural networks and their extensions for forecasting autoregressive causal-noncausal processes, characterized by local explosive patterns such as bubbles and spikes Process Definition and Characteristics The mixed noncausal autoregressive MAR(1,1) process is defined as the unique stationary solution of a specific equation involving errors with fat tails When the distribution of εt has fat tails, the parameters acquire an additional interpretation regarding spike and bubble rates The process contains both leads and lags of yt, admitting a two-sided moving average representation Network Architectures Compared The study compares the predictions of RNN, LSTM, GRU, Bidirectional LSTM, and a simple feed-forward network (FNN) to reveal the difficulty of forecasting extreme values depending on the type of network and the length of the training set The authors observe that generally, recurrent neural networks have difficulty predicting extreme values, and their forecast accuracy depends on the training set Forecasting Improvements through Methodology To improve the accuracy of point forecasts, the paper explores pooling of training sets and proposes to modify the loss function by replacing the mean square prediction error by using an alternative non-MSE criteria more adequate in the presence of bubbles The authors show that FNN outperforms other networks in predicting extreme values of a univariate noncausal time series during a bubble Loss Function Adaptation for Extremes The Mean Square Error is deemed a poor loss function for variables with possibly infinite moments, such as MAR(1,1) processes with Cauchy distributed errors Therefore, the loss function needs to be adjusted to accommodate the extremes The preferred method is a training loss function based on combining the regularized Student-t likelihood and the top 10 percent MAE Simulation Results and Conclusion The simulation results indicate that FNN is the best performing method both in group 1 and group 2 processes The presence of a bubble during the training period improves point predictions The best predictions are obtained from the FNN with a pooling mechanism that increases the training sets The authors conclude that the training loss function based on combining the regularized Student-t likelihood and the top 10 percent MAE is our preferred method, providing the best forecast of the series Appendix Details The second-order expansion of the asymptotic variance for a process with fat tails shows that in the under-linear regime (0 < β < 1), the variance is dominated by terms near the upper bound n ≈ T, and in the convergence regime (β > 1), it converges to a strictly positive constant C(β) as T → ∞ The optimal number of epochs k∗ is determined by the ratio NT = (1 − AT)(¯yT − 1t=10δ) − (B⊤ T Y + B10,T δ) and the learning rate coefficient β The timing of the shock matters for the bias: the closer to T, the larger its effect The optimal number of epochs k∗ is a constant, finite and greater than zero because individual update rate size µt is too fast to eliminate bias when β > 1 The analysis presented so far concerned a Gaussian process but the non-convergence of sample averages to the theoretical moments complicates the analysis further The choice of initial conditions becomes more impactful when dealing with heavy-tailed distributions Moreover, when sample averages are not consistent estimators of the theoretical moments, likelihood-type loss functions have to be used instead The timing of the shock matters for its size and sign in determining the optimal value k∗ The asymptotic behavior of k∗ depends on the learning rate coefficient β in µt = αt−β and on the shock δ The effect of the learning rate on the asymptotic behavior of k∗ is determined by the denominator ln(AT) When T → ∞ then by the Law of Large Numbers, y¯T → θ∗0, and (B′ T Y +B10,T δ) → (1−AT)θ∗0 As T → ∞, k∗ → 0 when 0 < β < 1 because the learning rate increases fast enough to overwrite the initial condition and shock effect so efficiently that no multiple epochs (k > 1) are required to minimize the error when the sample size tends to infinity When β = 1, k∗ → 0 logarithmically slow because multiple epochs are needed even for an infinitely large sample size When β > 1, k∗ is a constant, finite and greater than zero because the individual update rate size µt is too fast to eliminate bias The analysis presented so far concerned a Gaussian process but when a process with a heavy-tailed distribution is considered, the non-convergence of sample averages to the theoretical moments complicates the analysis further The choice of initial conditions becomes more impactful and finding optimal epoch numbers becomes more difficult Moreover, when sample averages are not consistent estimators of the theoretical moments, likelihood-type loss functions have to be used instead The timing of the shock matters for its size and sign in determining the optimal value k∗ When T → ∞ then by the Law of Large Numbers, y¯T → θ∗0, and (B′ T Y +B10,T δ) → (1−AT)θ∗0 As T → ∞, k∗ approaches zero when 0 < β < 1 because the learning rate increases fast enough to overwrite the initial condition and shock effect so efficiently that no multiple epochs (k > 1) are required to minimize the error when the sample size tends to infinity When β = 1, k∗ approaches zero logarithmically slow because multiple epochs are needed even for an infinitely large sample size When β > 1, k∗ is a constant, finite and greater than zero because the individual update rate size µt is too fast to eliminate bias The analysis presented so far concerned a Gaussian process but when a process with a heavy-tailed distribution is considered, the non-convergence of sample averages to the theoretical moments complicates the analysis further The choice of initial conditions becomes more impactful and finding optimal epoch numbers becomes more difficult Moreover, when sample averages are not consistent estimators of the theoretical moments, likelihood-type loss functions have to be used instead The timing of the shock matters for its size and sign in determining the optimal value k∗ When T → ∞ then by the Law of Large Numbers, y¯T → θ∗0, and (B′ T Y +B10,T δ) → (1−AT)θ∗0 As T → ∞, k∗ approaches zero when 0 < β < 1 because the learning rate increases fast enough to overwrite the initial condition and shock effect so efficiently that no multiple epochs (k > 1) are required to minimize the error when the sample size tends to infinity When β = 1, k∗ approaches zero logarithmically slow because multiple epochs are needed even for an infinitely large sample size When β > 1, k∗ is a constant, finite and greater than zero because the individual update rate size µt is too fast to eliminate bias

Title and authors: Ruth: So we're looking at this paper titled "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes." It seems like the authors are focusing on how these recurrent networks handle those local explosive patterns, things like bubbles and spikes.

Warren: That's right, Ruth. They're diving into autoregressive causal-noncausal processes, which is a specific kind of time series that has this structure of local explosions.

Iris: So the paper sets up these mixed noncausal autoregressive MAR(eleven) processes, and they note that when the error distribution has fat tails, you get an extra interpretation for those spike and bubble rates.

Ruth: Exactly. The key here is that this process has both leads and lags of the y t, meaning it can be represented by a two-sided moving average.

Warren: And they're comparing several network architectures, including RNNs, LSTMs, GRUs, Bidirectional LSTMs, and even a simple feed-forward network or FNN.

Iris: The main observation they make is that generally recurrent neural networks struggle to predict those extreme values that happen during a bubble.

Ruth: And the accuracy of their forecast really depends on the length of the training set they use.

Warren: To tackle this, they explore some specific improvements in their methodology, like pooling training sets and changing how they calculate the loss function.

Iris: They suggest replacing the mean square prediction error with an alternative non-MSE criteria that's better suited for dealing with bubbles.

Ruth: And the authors show that even a simple feed-forward network, the FNN, actually outperforms other networks when predicting extreme values in a univariate noncausal time series during a bubble.

Warren: So it seems like they found that sometimes simpler models can be more effective under these specific conditions.

Iris: The paper also suggests using the combination of the regularized Student-t likelihood and the top ten percent mean absolute error as their preferred training loss function.

Ruth: That sounds like a practical way to handle variables with potentially infinite moments, which is what those MAR(eleven) processes with Cauchy distributed errors are.

Warren: So to summarize the main findings from "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes," they look at how different recurrent network types perform when forecasting processes with local explosive patterns like bubbles and spikes.

Title and authors: Iris: And they point out that standard recurrent networks have difficulty predicting those extremes, and their performance is tied to having enough training data that includes those explosive patterns.

Ruth: They propose a couple of ways to make the predictions better: using pooling on the training sets and swapping the loss function for one that handles bubbles more effectively.

Warren: And they found that the FNN model was actually best in predicting extremes during a bubble, especially when they used that pooling mechanism to increase the size of their training sets.

Iris: They also concluded that combining a regularized Student-t likelihood with the top ten percent mean absolute error for their loss function gives them the best forecast of the series.

Ruth: So, in short, they found that for this type of process, using an FNN and that specific loss function helps get better point predictions when a bubble is happening.

Warren: It suggests that when you have these kinds of extreme patterns, you might not always need the most complex recurrent network architecture to get a good result.

Iris: The paper also touches on the technical details, like how the asymptotic variance behaves depending on whether the process is in an under-linear regime or a convergence regime.

Ruth: And it gets pretty deep into things like determining an optimal number of epochs, k*, based on learning rate coefficients and shock size delta.

Warren: It sounds like they are really digging into the math behind why certain training procedures work better than others for these heavy-tailed processes.

Iris: But there's a limitation they mention, which is that when sample averages aren't consistent estimators of theoretical moments, you have to switch to likelihood-type loss functions.

Ruth: That means if you can't trust your sample averages to represent the true underlying moments, the way you train the model has to change significantly.

Warren: So for someone just listening, it means that when dealing with these kinds of processes, you need a careful approach to both your model choice and how you're training it.

Title and authors: Iris: And that leads us into how this work might apply beyond just these specific time series models in the future.

Ruth: Well, this paper on "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes" suggests that for modeling things with bubbles and spikes, we need to be smart about our loss functions.

Warren: And maybe not always chasing the biggest recurrent architecture is the right move; sometimes a simpler model like an FNN can win when you're dealing with those specific types of extremes.

Iris: And they give us concrete tools, like that combination of likelihood and top ten percent MAE loss, which seems very robust for these heavy-tailed scenarios.

Ruth: It’s a study that really shows how adjusting the methodology can significantly improve forecasting accuracy when the data has those fat tails we often see in finance or other complex systems.

Warren: So, we've seen that even though recurrent networks have their limitations with extremes, you can get good results by tuning things like training set size and your loss function choice.

Iris: And the paper gives us a lot to think about regarding the identification of these explosive patterns and how they affect the estimation itself.

Ruth: It’s an important piece of work for anyone trying to build forecasting models for data that isn't perfectly Gaussian.

Warren: So, we wrap up our look at "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes." We see that tuning the training procedure, especially the loss function and training set size, makes a difference in how well these networks predict those volatile events.

Iris: And while they give us some clear methods for improvement, they also highlight where things get complicated when sample averages don't match theoretical moments.

Ruth: That’s the core of it; you need to be aware of both the wins and the caveats before applying these findings to your own work.

Warren: We’ll take this paper on "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes" and see how these ideas about loss function adaptation and model choice can help us build more stable systems.

The paper's summary: Ruth: So, to wrap up this paper on "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes," they're basically saying that these recurrent networks aren't great at predicting those extreme events like bubbles and spikes unless you tweak the training process.

Warren: They point out that standard recurrent architectures struggle with those fat-tailed processes, meaning their accuracy totally depends on how long the training set is.

Iris: The core of their argument is about adapting the loss function to handle those extremes, moving away from just mean square error because it doesn't work well when things have infinite moments.

Ruth: They suggest using a combination: mixing the regularized Student-t likelihood with the top ten percent mean absolute error. That should give you a better forecast of the series overall.

Warren: And they found that even a simple feed-forward network, an FNN, could perform better than those big recurrent networks when trying to predict extremes during a bubble.

Iris: That’s interesting because it suggests the complexity of the network might not be what drives the extreme value prediction in this specific setup.

Ruth: Exactly. And they showed that using a pooling mechanism on the training sets, which just means combining multiple smaller training sets, actually helps improve point predictions significantly.

Warren: So if you're trying to build a system that handles these volatile patterns, you need to think about both the model structure and how much historical data you feed it.

Iris: It’s a reminder that when dealing with heavy-tailed processes, the choice of training loss function is just as important as the architecture itself for getting reliable results.

Ruth: It really highlights that this isn't just about picking the fanciest network; it's about matching the math of your loss function to the nature of your data.

Warren: And they did touch on some deep technical stuff, like how the asymptotic variance behaves depending on whether you're in an under-linear regime or a convergence regime.

Iris: That means there are different mathematical behaviors depending on how fast the process is evolving over time, which complicates things for those who are just looking at simple averages.

Ruth: So what this means for us, as people trying to model complex systems, is that we need to be very careful about assumptions when things get explosive.

Warren: It’s a caution against assuming standard statistical tools work perfectly when you're dealing with these kinds of local shocks.

Iris: And the next part of this paper they look at involves figuring out the optimal number of training epochs, k*, which depends on factors like the learning rate coefficient beta.

Ruth: That shows that even in a well-defined process, you still have to optimize your training procedure based on parameters like how fast you are updating the model.

Warren: So we've seen some wins with this approach, but they also flag a limitation: if sample averages aren't consistent estimators of the theoretical moments, then likelihood-type loss functions become necessary.

Iris: That’s a crucial caveat; it tells us that we can’t always just rely on standard sample statistics when the underlying distribution is so heavy-tailed.

Ruth: So, while this paper offers practical tools for improving forecasting accuracy in bubble scenarios, the real challenge remains understanding exactly when those sample averages start failing us.

Warren: It sets up a clear path forward for tuning models: you need to balance model choice with data preparation and loss function design to handle these spikes.

Iris: And that leads us into the next part of this discussion, where we'll look at how this kind of methodology might apply to other sequential data problems outside of just these specific autoregressive processes.

The paper's improvements: Tom: So, we're talking about how they suggest fixing those problems in the paper on recurrent networks for these noncausal processes, focusing on specific technical upgrades to get better results.

Ruth: They really focus on changing the training loss function, moving away from just mean square error because it doesn't handle those heavy tails correctly.

Warren: They propose this combination: using the regularized Student-t likelihood and adding the top ten percent mean absolute error. That’s a concrete way to make sure you capture those extreme spikes.

Iris: I think that loss function modification is key because it directly addresses the issue of variables having possibly infinite moments in these processes.

Ruth: Exactly, it means you can get a more robust forecast of the series when there are bubbles happening.

Warren: And they also suggest using pooling for training sets, which just means combining multiple training sets to increase the total amount of data the network sees.

Iris: That helps improve point predictions by giving the model more context on what those extreme patterns look like across different parts of its history.

Ruth: It seems like a practical suggestion because it doesn't require you to overhaul your entire network architecture just to get better results in these specific scenarios.

Warren: And they also discuss using LSTM or GRU architectures instead of standard RNNs, which is about giving the network better memory through those gate mechanisms.

Iris: That’s a way to improve stability during training because it helps with more stable gradient propagation when dealing with long sequences.

Ruth: So the paper suggests a layered approach: tweak the loss function for tails, use pooling for data quantity, and maybe switch to an LSTM if you need better memory retention.

Warren: It’s about finding the right combination of methodological tweaks depending on what kind of extreme behavior you are trying to capture.

Iris: The implication is that even though recurrent networks have their limits with extremes, careful tuning of these specific components can yield much better performance.

Ruth: It really shows that for heavy-tailed data, the way you train the model is as important as the model structure itself.

Warren: So we’ve seen what they propose to fix things—better loss functions and more training data—but now we need to look at how those theoretical limits interact with these practical fixes.

Iris: The next thing they tackle is some deeper math about the asymptotic variance, looking at different regimes like the under-linear regime versus the convergence regime.

Ruth: That part gets into how fast the process is evolving over time and whether your model can keep up with that speed, which affects things like how many epochs you need to train for.

Warren: It’s a bit of an engineering problem here, figuring out the optimal number of epochs k* based on things like the learning rate coefficient beta.

Iris: That means if the process is converging fast enough, you might not need many epochs, but if it’s in a slow regime, you might need more updates to get it right.

Ruth: So these improvements give us better tools for forecasting volatility, but we still have that technical challenge of knowing exactly when those theoretical limits start to bite.

Warren: And that leads us right into the next topic: how this approach translates beyond just these specific autoregressive processes and what kind of real-world systems it might actually help predict.

Conclusion: Ruth: So, to wrap up this paper on "Recurrent Neural Networks for Forecasting Autoregressive Noncausal Processes," they’re essentially saying that getting good predictions for those explosive patterns requires a smarter way of training the AI model.

Warren: They found that tweaking the loss function, specifically using the Student-t likelihood and top ten percent mean absolute error, makes a big difference in how well you predict bubbles.

Iris: And they confirmed that pooling training sets helps improve point forecasts by giving the model more context on those extreme events across different parts of its history.

Ruth: It’s clear that for heavy-tailed data, the way we structure the loss function is just as important as choosing a complex recurrent architecture.

Warren: So, if you're building a strategy that needs to handle sudden spikes in volatility, you need to be thinking about both your model setup and your training objective.

Iris: I think their work on the asymptotic variance behavior is important because it shows the mathematical constraints we have when looking at these sequences over long periods.

Ruth: And they’ve flagged that when sample averages aren't consistent estimators of the theoretical moments, you’ve got to switch to likelihood-type loss functions, which is a big operational change for anyone using this approach.

Warren: It means we can't just rely on simple sample statistics when the underlying distribution is really heavy-tailed, so we have to be prepared to change our method if the moments don't match up.

Iris: So, the implication is that you need a flexible framework that can adapt its loss function based on how stable your sample averages are.

Ruth: That’s what it boils down to; adaptability in the training setup is what separates a good forecast from a bad one in these volatile markets.

Warren: This paper gives us concrete tools for handling those fat tails, but it also sets up that challenge of knowing exactly when your sample averages start failing you.

Iris: It’s a good reminder that when dealing with complex econometrics, you always need to check the identification—what variation is actually driving the signal.

Ruth: Well, we’ve looked at how this specific study on recurrent networks for noncausal processes works and what fixes they propose for handling extreme patterns.

Warren: It shows that tuning training procedures, especially the loss function and data pooling, makes a difference in how well these networks predict those volatile events.

Iris: It's a solid piece of work that provides clear methods for improvement when dealing with heavy-tailed processes in time series forecasting.

Ruth: We’ll keep an eye on how this idea of adaptive loss functions plays out in other complex financial modeling scenarios we look at next week.

Warren: And we’ll be looking at how these ideas about model choice and training tuning can help us build more stable systems for quantitative trading down the road.