E-backtesting

arXiv:2209.00991 · q-fin.RM, math.ST, stat.ME, stat.TH · Submitted 2022-08-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Quantitative Finance. The day in quantitative finance: models of markets, risk and prices.

Ruth: I'm Ruth.

Warren: And I'm Warren, here with Iris.

Ruth: We're looking at the paper: "E-backtesting".

Warren: Here's the gist of the paper. The paper introduces a model-free backtesting method utilizing e-values and e-processes to address challenges in backtesting risk measures, particularly Expected Shortfall (ES), which has become the standard risk measure under recent Basel Accords.

Ruth: The primary contributions are multifaceted: first, defining backtest e-statistics and proposing novel backtesting methods, yielding specific statistics for ES (Theorem 1) and other risk measures (Theorem 2); second, proposing three data-based betting process calculations—GRO, GREE, GREL, and GREM—which are shown to be asymptotically optimal in various scenarios (Theorem 3); third, characterizing the backtest e-statistics for mean, variance, VaR (Theorem 4), and ES (Theorem 5); and finally, demonstrating detailed procedures for backtesting VaR and ES using e-values in practical financial operations.

Warren: The methodology centers on constructing a stochastic process M t defined by M 0 = 1 and M t = (1 - lambda t + lambda t X t) M t-1 = Y s = (1 - lambda s + lambda X s), for t in

T: .

Ruth: Backtest e-statistics are defined, and the paper establishes that for ES, the function e ES p is a monotone backtest e-statistic for (ES p, VaR p) (Theorem 1).

Title and authors: Ruth: Okay, so we’ve seen the basic setup of E-backtesting and how they’re defining their statistics. Now let’s look at what exactly this paper is saying in its summary about the methods they developed and why it matters to us right now >

Warren: Basically, the core idea is building a stochastic process called M t starting at one, where M t depends on some lambda and X t, which represents the actual losses or risk factors over time >

Iris: They show that for ES specifically, this function e ES p is a monotone backtest e-statistic for both Expected Shortfall and Value-at-Risk simultaneously using theorem one >

Ruth: That monotonicity is important because it means as you test, the statistic moves in a predictable way, which simplifies things when you're trying to see if a risk measure is being underestimated or overestimated >

Warren: The paper also discusses the choice of betting process being crucial, and they propose GRO for growth-rate optimal, GREE for empirical e-statistics, GREL for empirical losses, and GREM for empirical mixture >

Iris: They show that GREM is recommended as a stable default because it performs well across different testing situations >

Ruth: It sounds like they’re giving practitioners a clear path on which betting process to choose depending on what kind of data or risk scenario they're facing when backtesting ES >

The paper's summary: Warren: Moving past the basic summary, the authors point out several real improvements they are proposing for this whole E-backtesting approach, which I think are pretty significant for how we actually use these tools >

Iris: One big improvement is that you can perform backtest ES for market risk using model-free e-values and e-processes, which lets you test underestimation of tail risk without needing to have a specific distributional forecast >

Ruth: That’s really useful because it allows regulators to get alerted early because the "e-process accumulates to a reasonably large value," rather than waiting for some complex model output >

Warren: They also suggest using the GREM method for choosing the betting process, which they propose as an average of GREE and GREL, providing a robust approach when there isn't a clear pattern in losses or forecasts >

Iris: They suggest that the e-backtesting procedure can be used with mean-variance criteria to determine portfolio weights, and this is interesting because it doesn't require you to know any specific trading strategy or statistical model for that >

Ruth: So, the idea is making the process flexible enough that you don't need a perfect model of the underlying economy just to do a decent risk check >

The paper's improvements: Ruth: We’ve gone over how they defined their statistics and which betting processes are best for ES, but let’s wrap up with what this all means for the practical application in the real world >

Warren: The main implication is that e-backtesting is presented as the first model-free and non-asymptotic backtest for Expected Shortfall, which means it's something you can actually run without needing a fixed sample size or assuming a specific model structure on your data >

Iris: They also characterized the necessary conditions for the existence of backtest e-statistics, revealing that for risk measures like ES, the functional has to be quasi-linear >

Ruth: So while this method offers validity across all stopping times and feasibility without underlying model assumptions, they admit a trade-off where these e-backtests might be less powerful than traditional model-based approaches >

Warren: They also pointed out that the characterization results identify essentially unique optimal choices for backtest e-statistics for VaR and Expected Shortfall, which is good because it helps narrow down the search space for us >

Iris: And finally, they showed detailed procedures for practical use in financial operations, which is where we get to see how this actually works on a portfolio level >

Ruth: So the paper E-backtesting really gives us a robust way to check if our risk forecasts are underestimating tail risk without having to commit to a specific distributional assumption on the losses >

Conclusion: Ruth: So we’ve seen how they’re building this model called E-backtesting, and what it means for testing if risk measures like Expected Shortfall are being underreported >

Warren: Right, so you're saying it lets us test if tail risk is being underestimated without having to guess the exact shape of the loss distribution >

Iris: That's the core idea. They show this function e ES p is monotone for both expected shortfall and value at risk simultaneously, which means we can track how well our forecasts are working over time >

Ruth: And they’ve got these betting processes like GRO, GREE, GREL, and GREM to choose from; they even recommend GREM as a stable default because it handles different testing situations well >

Warren: The authors also show that you can use this method to detect evidence against forecasts with thresholds of two or five or ten on the e-process value >

Iris: That early warning mechanism is pretty strong, much better than just doing some traditional p-tests, because it watches the accumulation of risk evidence >

Ruth: It’s interesting how they talk about institutional gaming here; they show that using a properly calculated betting process like GREM can help institutions stop under-reporting risks >

Warren: They suggest that GREM is superior in that context because it keeps the e-process near one during over-reporting and grows fast during under-reporting, unlike some of the others >

Iris: It’s important to remember they also characterized the necessary conditions for these statistics to exist, showing that for risk measures like ES, you need a quasi-linear functional >

Ruth: So while this method is flexible—it works across all stopping times and doesn't require an underlying model—they admit it might not be as powerful as some of the traditional model-based approaches >

Warren: It’s a trade-off, I guess. But they also point out that their characterization results help us find the most optimal choices for these backtest statistics for value at risk and expected shortfall >

Iris: Exactly. And in terms of practical application, they showed how you can use this e-backtesting procedure with mean variance criteria to determine portfolio weights without needing any specific trading strategy >

Ruth: It’s pretty broad then, moving beyond just a specific financial product to general portfolio risk management using these principles >

Warren: So the takeaway is that E-backtesting gives us a way to check risk underestimation robustly, even when we don't have perfect models of the losses themselves >

Iris: It’s still important to remember the authors flag that this method doesn't replace a full model-based approach; it supplements it by checking for estimation errors >

Ruth: That’s right. So we have this paper, "E-backtesting," showing a new way to test risk without needing perfect assumptions about the loss distribution >

Warren: It’s a practical tool that helps spot when institutions might be hiding tail risks in their reports >

Iris: Next time, we’ll look at how AI is actually messing with market data and where those outages are hiding things.

Qiuqi Wang, Ruodu Wang, Johanna Ziegel

Georgia State University · University of Waterloo · ETH Zurich

q-fin.RM, math.ST, stat.ME, stat.TH

Submitted: 2022-08-27

Updated: 2026-11-06

Comments: 68 pages

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 100/100

The gist: The paper introduces a model-free backtesting method utilizing e-values and e-processes to address challenges in backtesting risk measures, particularly Expected Shortfall (ES)…

Key concepts

Backtest e-statistics
These are statistical measures derived from a stochastic process used to test if a risk forecast is accurate. The paper defines specific backtest e-statistics for ES and other risks, showing how they behave when testing predictions.
Betting Processes (GRO, GREE, GREL, GREM)
These are different ways to calculate the underlying process used in backtesting. They are designed to capture different aspects of risk behavior—like growth rate or empirical losses—and GREM is recommended for stability.
e-values and e-processes
These are tools that allow for model-free backtesting. They help detect evidence against forecasts by accumulating values in an e-process, providing an early warning signal when risk estimates might be underestimated.

Terminology

Summary

The paper introduces a model-free backtesting method utilizing e-values and e-processes to address challenges in backtesting risk measures, particularly Expected Shortfall (ES), which has become the standard risk measure under recent Basel Accords. p. 0

Where the paper says it

E-backtesting Qiuqi Wang∗ Ruodu Wang† Johanna Ziegel‡ 16th April 2026 Abstract In the recent Basel Accords, the Expected Shortfall (ES) replaces the Value-at-Risk (VaR) as the standard risk measure for market risk in the banking sector, making it the most important risk measure in financial regulation. One of the most challenging tasks in risk modeling practice is to backtest ES forecasts provided by financial institutions.

The primary contributions are multifaceted: first, defining backtest e-statistics and proposing novel backtesting methods, yielding specific statistics for ES (Theorem 1) and other risk measures (Theorem 2); second, proposing three data-based betting process calculations—GRO, GREE, GREL, and GREM—which are shown to be asymptotically optimal in various scenarios (Theorem 3); third, characterizing the backtest e-statistics for mean, variance, VaR (Theorem 4), and ES (Theorem 5); and finally, demonstrating detailed procedures for backtesting VaR and ES using e-values in practical financial operations. p. 2
Where the paper says it

It turns out that these methods are asymptotically optimal (equivalent to an oracle betting process) in different situations (Theorem 3). Third, we characterize backtest e-statistics for the mean, the variance, VaR (Theorem 4), and ES (Theorem 5) in Section 6.

The methodology centers on constructing a stochastic process M t defined by M 0 = 1 and M t = (1 - lambda t + lambda t X t) M t-1 = Y s = (1 - lambda s + lambda X s), for t in [T]. p. 5

Where the paper says it

(iv) Construct a stochastic process: M0 = 1 and Mt = (1 − λt + λtXt)Mt−1 = Y t s=1 (1 − λs + λsXs), t ∈ [T]. (v) Reject H0 at time t if we observe Mt > 1/α.

Backtest e-statistics are defined, and the paper establishes that for ES, the function e ES p is a monotone backtest e-statistic for (ES p, VaR p) (Theorem 1). p. 2
Where the paper says it

Third, we characterize backtest e-statistics for the mean, the variance, VaR (Theorem 4), and ES (Theorem 5) in Section 6. All backtest e-statistics for these functionals take similar forms as mixtures between the constant function equal to 1 and a simple backtest e-statistic.

The choice of betting process is crucial; methods like GRO (growth-rate optimal), GREE (growth-rate for empirical e-statistics), GREL (growth-rate for empirical losses), and GREM (growth-rate for empirical mixture) are proposed, with GREM recommended as a stable default due to its performance.

Simulation studies validate the method's advantages, showing that evidence against forecasts is detected using thresholds of 2, 5, or 10. The e-backtesting procedure is presented as the first model-free and non-asymptotic backtest for ES. p. 1

Where the paper says it

Table 1 summarizes the main features of existing methods backtesting ES. To the best of our knowledge, there is no model-free non-asymptotic backtesting method for ES.

Furthermore, necessary conditions for the existence of backtest e-statistics are characterized (Proposition 4), revealing that for risk measures like ES, the functional must be quasi-linear. p. 0
Where the paper says it

To design a model-free backtesting procedure for ES, we make use of the recently developed techniques of e-values and e-processes. Backtest e-statistics are introduced to formulate e-processes for risk measure forecasts, and unique forms of backtest e-statistics for VaR and ES are characterized using recent results on identification functions.

While e-backtests offer validity across all stopping times and feasibility without underlying model assumptions, a trade-off exists: they may be less powerful than traditional model-based approaches. p. 1
Where the paper says it

Fourth, the tester does not necessarily accurately know the underlying model used by a financial institution to produce risk predictions. In financial practice, a well-adopted simple approach exists for backtesting the Value-at-Risk (VaR), which is the so-called three-zone approach based on binomial tests described in BCBS (2013); this approach is model-free in the sense that one directly tests the risk forecast without testing any specific family of models.

The characterization results identify essentially unique optimal choices for backtest e-statistics for VaR and (ES, VaR). p. 0
Where the paper says it

To design a model-free backtesting procedure for ES, we make use of the recently developed techniques of e-values and e-processes. Backtest e-statistics are introduced to formulate e-processes for risk measure forecasts, and unique forms of backtest e-statistics for VaR and ES are characterized using recent results on identification functions.

A significant practical application demonstrated is addressing the issue of financial institutions gaming regulators by producing conservative forecasts initially and then under-reporting risks. This is mitigated by using a properly calculated betting process. p. 3

Where the paper says it

Five sections are put in the e-companion. Section A shows the betting processes calculated via Taylor approximation for VaR and ES.

The GREM method proves superior in this context; it guarantees that the e-process stays near 1 during over-reporting periods and grows quickly during under-reporting, unlike GREE or GREL which can exhibit different behaviors (e. p. 1
Where the paper says it

As a particularly relevant feature to our context, our proposed e-tests allow regulators to get alerted early as the e-process accumulates to a reasonably large value. This is different from scientific discoveries (such as genome studies) where a scientist may not be entitled to reject a 1Quoting BCBS (2016, p.

g., GREE's log e-process staying close to 0 during over-reporting). The use of a rolling window (e. p. 0
Where the paper says it

One of the most challenging tasks in risk modeling practice is to backtest ES forecasts provided by financial institutions. To design a model-free backtesting procedure for ES, we make use of the recently developed techniques of e-values and e-processes.

g., 250 or 500 days) instead of all historical data in calculating the betting process helps manage the impact of past over-prediction, suggesting that GREM provides a robust solution due to its sequential testing nature.

The forecasting procedure involves estimating model parameters using standard Gaussian QML on presampled losses, followed by forecasts for VaR and ES based on empirical forecasts derived from presampled residuals under various innovation distributions (normal, t-, or skewed-t). p. 0

Where the paper says it

To design a model-free backtesting procedure for ES, we make use of the recently developed techniques of e-values and e-processes. Backtest e-statistics are introduced to formulate e-processes for risk measure forecasts, and unique forms of backtest e-statistics for VaR and ES are characterized using recent results on identification functions.

The paper details simulations involving both stationary time series and structural change scenarios, including a burn-in period. p. 2
Where the paper says it

Parametric or dependence assumptions refer to those on loss distributions, time series models, stationarity, or strong mixing. Forecast structural assumptions refer to requirements on the forms and properties of risk forecasts.

For the final sample used in backtesting (4280 negated percentage log-returns), the portfolio assumptions are tested against normal, t-, or skewed-t distributions using maximum likelihood estimation for an AR(1)-GARCH(1, 1) process. The overall conclusion is that e-backtesting is highly useful in realistic settings where sample size is not fixed or no models are explicitly tested alongside risk measure forecasts, and the methodology can be extended to more general risk measures. p. 2
Where the paper says it

Table 1: Comparison of existing backtesting methods for ES Literature Parametric or dependence assumptions Forecast structural assumptions Fixed sample size Asymptotic test Reliance on VaR or distributional forecasts MF00 yes yes yes yes yes AS14 yes yes yes yes yes DE17 yes yes yes yes yes NZ17 yes yes yes yes yes SQPQ21 yes yes yes yes yes BD22 yes yes yes yes no HD22 yes yes no no yes This paper no no no no yes Notes: We use shortcuts MF00 for McNeil and Frey (2000), AS14 for Acerbi and Sz´ekely (2014), DE17 for Du and Escanciano (2017), NZ17 for Nolde and Ziegel (2017), SQPQ21 for Su et al. (2021), BD22 for Bayer and Dimitriadis (2022), and HD22 for Hoga and Demetrescu (2023).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Backtest ES for market risk can be performed using model-free e-values and e-processes, which allows for testing under estimation of tail risk rather than requiring specific distributional forecasts. This enables regulators to get alerted early as the e-process accumulates to a reasonably large value.

  2. The AI system can utilize the GREM method for choosing the betting process, which is proposed by taking the average of the GREE and GREL methods, providing a robust approach when there is no clear pattern in losses or forecasts.

  3. The system can detect evidence against risk forecasts with high probability using thresholds 2, 5, or 10 for e-processes exceeding these values, offering an early warning mechanism superior to traditional p-tests.

  4. The AI system can monitor structural changes in time series data by comparing the results of the e-backtesting method with the sequential monitoring method, which is shown to outperform the e-backtesting methods in terms of detection percentage and speed.

  5. For portfolio risk management, the system can apply the e-backtesting procedure using mean-variance criteria to determine portfolio weights, without requiring knowledge of a specific trading strategy or statistical model.

Abstract

In the recent Basel Accords, the Expected Shortfall (ES) replaces the Value-at-Risk (VaR) as the standard risk measure for market risk in the banking sector, making it the most important risk measure in financial regulation. One of the most challenging tasks in risk modeling practice is to backtest ES forecasts provided by financial institutions. To design a model-free backtesting procedure for ES, we make use of the recently developed techniques of e-values and e-processes. Backtest e-statistics are introduced to formulate e-processes for risk measure forecasts, and unique forms of backtest e-statistics for VaR and ES are characterized using recent results on identification functions. For a given backtest e-statistic, a few criteria for optimally constructing the e-processes are studied. The proposed method can be naturally applied to many other risk measures and statistical quantities. We conduct extensive simulation studies and data analysis to illustrate the advantages of the model-free backtesting method, and compare it with the ones in the literature.

Sources

Related papers