Backtesting High-Dimensional Expected Shortfall

summary

The gist

This paper develops out-of-sample Expected Shortfall (ES) backtests designed to handle high-dimensional estimation of Value-at-Risk (VaR), ES, and the evaluation instrument.

In short

This research develops out-of-sample backtests for Expected Shortfall (ES) when estimating it in high dimensions, addressing challenges related to estimating both the tail threshold and conditional mean. The authors introduce orthogonal backtesting moments using a residualized test function to remove estimation bias, establishing oracle equivalence for valid inference under complex estimation conditions.

Key concepts

Expected Shortfall (ES) Backtest
A statistical test used to check if an estimated ES model accurately predicts actual losses. The paper focuses on high-dimensional settings where estimating the true risk is difficult because it requires estimating both a threshold and a conditional mean, which introduces estimation errors.
Orthogonal Backtesting Moments
A method developed to correct standard backtests by multiplying the ES residual with a residualized test function. This correction removes bias caused by how the Value-at-Risk (VaR) and ES models are estimated, making the test more robust against estimation errors.
Oracle Equivalence
This concept proves that under certain conditions—like dependent observations or overlapping estimation windows—the backtest statistic behaves as if the true population parameters were known. This ensures that even when models are repeatedly updated, the statistical inference remains valid.
Residualized Test Function
A component used in the orthogonal correction. It is a function derived from residuals that helps isolate and remove the portion of the ES residual that is simply reacting to local changes or errors in estimating the ES model itself.

Terminology used across episodes

This episode discusses

The paper

Backtesting High-Dimensional Expected Shortfall · Read on arXiv

Zaichao Du, Juan Carlos Escanciano, Pei Pei

Fudan University · Universidad Carlos III de Madrid · Central University of Finance and Economics

Transcript

Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "Backtesting High-Dimensional Expected Shortfall".

Warren: Here's the gist of the paper. This paper develops out-of-sample Expected Shortfall (ES) backtests designed to handle high-dimensional estimation of Value-at-Risk (VaR), ES, and the evaluation instrument.

Ruth: A primary challenge addressed is that ES forecasts depend on both an estimated tail threshold and an estimated conditional tail mean.

Warren: The first major contribution introduces a characterization of orthogonal backtesting moments within a class derived from VaR and ES residuals.

Ruth: This leads to a simple correction: multiplying the transformed ES residual by a residualized test function, which effectively removes the component of the residual responding to local changes in the ES model.

Title and authors: Ruth: So we’re talking about the paper "Backtesting High-Dimensional Expected Shortfall." It looks like they're tackling a big problem with how we check these complex tail-risk models.

Warren: Right. It focuses on out of sample expected shortfall backtests that are designed to handle high dimensional estimation of value at risk, expected shortfall, and the evaluation instrument. That sounds complicated, but what's the actual issue they’re trying to solve?

Iris: The main difficulty they pinpoint is that ES forecasts rely on two things: an estimated tail threshold and an estimated conditional tail mean. So you can't just test one thing in isolation; you have to account for both of those estimations being imperfect.

Ruth: Exactly. And the first big contribution they make is characterizing some orthogonal moments within a class derived from value at risk and expected shortfall residuals. This leads to this simple correction where you multiply the transformed ES residual by a residualized test function.

Warren: So, what does that correction actually do in practice? Does it just make the number look bigger or smaller?

Iris: It effectively removes the part of the residual that is responding to local changes in that ES model itself. If you're using linear expected shortfall specifications, they show you can implement this correction using a three step Double Lasso regression with a tail size adjusted score as the dependent variable.

Ruth: That’s pretty concrete. And then they move on to showing oracle equivalence for backtests involving three estimated components: value at risk, expected shortfall, and that residualized test function. This holds under conditions of dependent observations and overlapping or expanding estimation windows.

Warren: Oracle equivalence is a strong claim. What does that mean for us as practitioners who are constantly retraining these models on new data?

Iris: It means when the combined effect of changing value at risk, expected shortfall, and the evaluation instrument is asymptotically negligible, the feasible statistic converges to the same limiting distribution as if we actually knew those population components. That should help us trust our backtests even when our models are constantly evolving.

Ruth: It’s about establishing valid inference under these conditions while preserving your original forecasting procedures. They connect two different literatures here: ES backtesting procedures using realized tail losses and cumulative violations, or building on the joint elicitability of value at risk and expected shortfall.

Warren: That sounds like they are trying to bridge a gap between how we test these things practically and the theoretical framework that underpins them. What about the empirical side of things? Do they just show theory?

Iris: They do show an empirical application to downside growth forecasting for G7 economies, and they found evidence against a US growth only expected shortfall specification even when the forecast oriented regularization selects no additional financial predictors.

Ruth: That’s really important because it highlights that predictive variable selection and backtesting serve different roles in evaluating high dimensional tail risk models. They aren't interchangeable things.

Warren: So, what does this mean for someone who is just trying to build a better growth forecast model? Does this paper tell me which variables to pick?

Iris: It suggests that predictive variable selection cannot substitute for backtesting and must account for those estimation effects we talked about. The broader contribution is separating two sources of difficulty in tail risk model review: economically relevant departures from an expected shortfall specification versus the estimation effects that can distort the evidence against it.

Ruth: So, by using these orthogonal backtests, you mitigate leading estimation effects while still allowing your test functions to target substantive departures of interest. And they found that accounting for estimation can actually alter rejection decisions in some cases, like in Canada.

Warren: That’s interesting because it means the correction isn't just a mathematical trick; it has real-world consequences on whether we reject or fail to reject a model. So, if you look at the final result of "Backtesting High-Dimensional Expected Shortfall," what does that tell us about size control?

Iris: The paper concludes that orthogonalization improves size control relative to baseline expected shortfall backtests. Simulations indicate that conventional backtests can severely overreject optimal forecasts, whereas orthogonalization enhances size control.

Ruth: So we have a method that makes the tests more controlled without losing the ability to find real misspecification of the ES model itself. It’s about improving how much confidence we get out of those backtests.

Warren: It sounds like a solid piece for anyone dealing with high dimensional forecasting where you're worried about overreacting to noise in your testing process. So, before we wrap up this discussion on "Backtesting High-Dimensional Expected Shortfall," Iris, what’s the one final thing you want us to take away?

Iris: I think the most crucial point is that this method provides interpretable directions for model revision by allowing flexible estimation of value at risk to isolate expected shortfall misspecification from quantile model misspecification.

Ruth: And I'd add that it confirms that you can use residualized instruments to retain information in omitted predictors even when they aren't explicitly fitted in the ES model, which is a practical way to find relevant omitted information.

Warren: So we’ve covered how these orthogonal backtests help control size, how oracle equivalence holds under shifting windows, and how they separate estimation errors from true specification errors. That’s a lot of machinery for high dimensional risk assessment.

The paper's summary: Ruth: So we’re looking at how this paper boils down to its main points about checking these complex tail risk models and what that means for us in practice.

Warren: Basically, they're dealing with the fact that expected shortfall forecasts depend on two things—the threshold you pick and the average loss in the tail—and both of those are estimates, so you can't just test one thing by itself.

Iris: And their main contribution is a way to fix that by introducing a residualized test function. You multiply the transformed expected shortfall residual by that function, which basically cancels out the part of the error that comes from how poorly you estimated the ES model itself.

Ruth: That makes it less sensitive to how much you messed up with your VaR estimate or your conditional tail mean estimate. They show this correction works even for linear expected shortfall specifications using a three step Double Lasso regression.

Warren: So, they’re giving us a simple way to make the backtest more robust against the noise in our initial model fitting process. That sounds good for reducing false alarms caused by estimation error, right?

Iris: Exactly. And they prove something called oracle equivalence, which is pretty important because it means if you use this method with overlapping or expanding windows of data, your test statistic behaves like it would if you actually knew the true underlying population parameters.

Ruth: That's a big deal for anyone running rolling forecasts, because it validates your tests even when the model keeps getting retrained. It shows that as long as the combined effect of changing all those components isn't too big, you can trust your results.

Warren: But what about the real-world application? They tested this on downside growth forecasting for G7 economies and found that even when the model selection process picks no extra predictors, their backtest could still catch issues with a US growth only expected shortfall specification.

Iris: That points to a key difference between variable selection and specification testing. You can select variables, but you still need the backtesting part to check if your ES assumption is wrong, not just if you picked the right predictors for forecasting.

Ruth: So the paper’s core message is separating two kinds of problems in reviewing tail risk models: whether something fundamentally wrong with how you modeled the expected shortfall itself, or just whether your estimation process messed up the evidence against it.

Warren: And they show that their method helps mitigate those estimation errors while letting you still target actual model misspecification, which actually changed some rejection decisions in a study in Canada.

Iris: They also highlighted that this orthogonalization improves size control compared to standard ES backtests; conventional methods often overreject the right forecasts, but this correction makes the tests more controlled.

Ruth: So for anyone working on high-dimensional downside risk, it seems like this method offers a way to get tighter controls on your backtest while still being able to find those substantive errors you're actually looking for.

Warren: It sounds like a solid tool for handling the complexity of testing these models when you can’t assume everything is perfectly estimated.

The paper's improvements: Ruth: So now we’re looking at the actual improvements they suggest to make this backtesting method even better than what we just discussed and what that means for how we use it day-to-day.

Warren: Basically, they're talking about how to fix the estimation issues on their own model—the VaR and ES parts—to make sure those corrections work even smoother.

Iris: They propose this three step Double Lasso regression using a tail size adjusted score as the dependent variable. This lets you isolate whether your misspecification is coming from the quantile model or from how you’re handling the actual conditional tail risk.

Ruth: That gives us a much better tool for revision because we can pinpoint exactly where the model is failing, instead of just getting a vague error signal across all components.

Warren: It sounds like they're giving us more granular control over our model tuning process and not just throwing everything into the Lasso box.

Iris: And they also talk about oracle equivalence under these shifting data conditions, meaning their method remains reliable even when you’re constantly retraining your models on new, overlapping sets of observations.

Ruth: That's a huge practical win for dynamic forecasting because it means you don't have to re-validate your whole backtest every single time the data window moves.

Warren: So, it sounds like the method is designed not just to pass a test once but to survive a whole process of constant updating without losing its accuracy.

Iris: And they emphasize that this allows for flexible estimation, including machine learning methods, so you can use whatever tools you have to find the right combination of VaR and ES estimates.

Ruth: That gives practitioners more power to customize their testing procedure instead of being stuck with just one fixed mathematical formula.

Warren: So, it’s about giving the user more agency in how they test their risk models while maintaining a high standard of size control.

Iris: And they also show that this approach can actually help you find omitted predictors, meaning there might be other important information in your data that you missed because your fitted ES model didn't include it.

Ruth: That’s interesting because it means the backtest isn't just checking what’s in the model; it can point us toward variables we should consider adding to the model entirely.

Warren: So, this isn't just about fixing a number; it’s about improving how we build and maintain our entire downside risk evaluation system.

Conclusion: Tom: So to wrap things up, this paper "Backtesting High-Dimensional Expected Shortfall" is essentially about building a more robust way to test complex tail risk models by separating estimation errors from actual model mistakes.

Ruth: It shows that by using orthogonalization, you can get better size control on your backtests while still being able to find the real flaws in your expected shortfall assumption.

Warren: Yeah, so the core idea is making sure that when you check your forecast, you’re not just getting a false alarm because your initial model didn't fit perfectly.

Iris: I think the oracle equivalence part is key here because it guarantees that even with constantly updated models and overlapping data windows, the test statistic still points to the right conclusion if everything is asymptotically negligible.

Ruth: It’s practical for anyone running a rolling forecast because it means your tests don't become totally unreliable as you update your inputs.

Warren: That makes sense, though I gotta ask, this method doesn't help if the underlying economic relationship itself is fundamentally wrong, right?

Iris: The paper makes that clear; they separate estimation effects from specification errors, which means you can still find substantive departures even if the estimation process is noisy.

Ruth: So the implication for us is that we don't have to choose between a perfect model fit and a reliable backtest procedure anymore.

Warren: It’s about getting both of those things done simultaneously without one totally sabotaging the other, right?

Iris: Exactly. And they show this approach can even help you identify omitted predictors in your data that might be relevant for tail risk but just weren't in your fitted model.

Ruth: That’s a useful direction because it gives us a way to look beyond what we thought was important when we set up our initial ES formula.

Warren: So, it’s about making the whole process more transparent and less prone to those hidden pitfalls that show up when you have high dimensions.

Iris: In short, "Backtesting High-Dimensional Expected Shortfall" gives us a more rigorous way to evaluate these models under messy real-world data conditions.

Ruth: It’s a solid piece of work for anyone dealing with complex risk forecasting, and it sets a new standard for how we should be checking these things.

Warren: I think this paper paves the way for more reliable high dimensional tail risk analysis, which is something we need as AI models get more integrated into our decision-making.

More episodes

← Home