Testing Clustered Equal Predictive Ability with Unknown Clusters
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Testing Clustered Equal Predictive Ability with Unknown Clusters".
Warren: Here's the gist of the paper. We develop tests of clustered equal predictive ability (C-EPA) in panel data where clusters are unknown and must be estimated from the data using the Panel Kmeans algorithm.
Ruth: To address the challenge of testing hypotheses dependent on these data-driven clusters, we propose a selective conditional inference framework designed to mitigate the "double dipping" problem inherent when clustering is used in both hypothesis testing and cluster estimation.
Warren: The core methodology involves deriving a Wald-type test for pairwise equality of cluster centers, demonstrating that its limiting distribution, conditional on the estimated clusters, is that of a truncated chi variable.
Ruth: We then propose a p-value combination method by aggregating evidence against both the pairwise equality null hypotheses and the overall Equal Predictive Ability (EPA) null hypothesis.
Title and authors: Ruth: To wrap up what we just heard about "Testing Clustered Equal Predictive Ability with Unknown Clusters," the paper essentially introduces a method for testing equal predictive ability in panel data when you don't know the clusters that generate that predictive ability.
Warren: They use Panel Kmeans to estimate these unknown clusters, and then they propose a selective conditional inference framework to test hypotheses about clustered equal predictive ability by conditioning on those estimated clusters.
Iris: It’s addressing the problem of double dipping where you use data-driven clusters both for estimation and testing, which is a known issue in this literature.
Ruth: Specifically, they derive a Wald-type test for pairwise equality and show that its square root conditional on the estimated clusters follows a truncated chi variable distribution.
Warren: And then for the overall equal predictive ability hypothesis, they use a p-value combination method by aggregating evidence from both the pairwise equality tests and the overall EPA null hypothesis.
Iris: The main statistic they derive is FSI r, which allows them to combine these pieces of evidence under some arbitrary dependence among the input p-values by selecting a value of r within a specific range.
Ruth: The authors are also looking at how to select the number of clusters using an information criterion, IC(K), instead of relying on something like Bonferroni correction, which they say is computationally more efficient and avoids conditioning on the choice of the criterion itself.
Warren: And they show that using multiple Kmeans initializations in their simulations doesn't invalidate their procedure, so they’re checking for robustness there as well.
Iris: The paper also examines the role of multiple Kmeans initializations in simulations to see if the result holds up under different starting points for the clustering algorithm.
The paper's summary: Ruth: So, summarizing what they actually did, this paper develops tests of clustered equal predictive ability when clusters are unknown and must be estimated using a method like Panel Kmeans.
Warren: The core contribution is moving beyond the assumption that we know the clusters beforehand by proposing a selective conditional inference framework to handle these data-driven clusters.
Iris: They first build a Wald-type test for pairwise equality and show its limiting distribution when conditioned on the estimated clusters is that of a truncated chi variable, which helps control Type one errors.
Ruth: Then, they propose combining this with an overall equal predictive ability test using a p-value merging function valid under dependence among the components to get their main statistic FSI r.
Warren: The authors also show that selecting the number of clusters via an information criterion, IC(K), is fine and doesn't break the proposed procedure.
Iris: The central inferential difficulty they address is that conventional tests treat these data-driven clusters as fixed, which causes spurious evidence of heterogeneity due to this post-selection problem.
Ruth: Their solution is to use the full sample while conditioning on the clustering outcome, which avoids relying on sample splitting or external cluster information.
Warren: The paper confirms their method works well in simulations, showing accurate size control and good finite-sample power for both the pairwise equality tests and the overall EPA test.
Iris: Their empirical application to exchange-rate forecasting shows that this procedure is practically relevant because it reveals heterogeneous predictive gains missed by aggregate tests when latent clusters are accounted for.
The paper's improvements: Ruth: One of the main improvements they suggest is replacing naive post-clustering tests with their selective inference procedures that account for the data-dependent nature of cluster formation via truncation based on the estimated clusters.
Warren: That’s a direct fix for the double dipping issue, and it moves away from relying on external information or sample splitting to control Type one error rates.
Iris: They also developed a test statistic FSI,r that combines pairwise equality tests with an overall equal predictive ability test, providing a valid p-value under arbitrary dependence among the input p-values by selecting a value of r within the admissible range of negative values.
Ruth: And they introduced the information criterion for estimating the number of clusters, IC(K), which is computationally cheaper than cross-validation and avoids conditioning on which specific criterion you choose.
Warren: That's a practical improvement because it makes choosing how many clusters to estimate less dependent on an arbitrary choice of an external selection tool.
Iris: They also showed that the square root of the Wald-type statistic has an asymptotic truncated chi distribution conditional on the selected clustering outcome, which is a very specific distributional result they want to prove.
Ruth: So, these improvements are all aimed at making sure that this test isn't just theoretically sound but also practically usable for controlling size and maintaining power in real applications.
Conclusion: Warren: So to wrap up the "Testing Clustered Equal Predictive Ability with Unknown Clusters," the paper proves that their selective conditional inference test, FSI r, controls the type one error rate asymptotically and remains consistent even when either the homogeneity null or the overall equal predictive ability null fails.
Iris: That means this framework is a robust way to handle unknown clusters without needing prior knowledge of them, because it accounts for model selection uncertainty internally.
Ruth: The final conclusion is that this selective conditional inference test offers a method that controls size effectively and is preferred over naive post-clustering tests because it directly addresses how data-driven cluster formation affects the inference.
Warren: They confirmed this with empirical work in exchange-rate forecasting, showing that accounting for latent clusters shows heterogeneous predictive gains missed by aggregate tests, which really validates the approach.
Iris: I think what this means for us is that when we look at panel data, we should stop looking for simple aggregate effects and start conditioning on the cluster structure estimated from the data itself if those structures matter.
University of Bourgogne Europe · Paris-Panthພon-Assas University · Bayes Business School (formerly Cass) · School of Economics, Singapore Management University
econ.EM
Submitted: 2025-07-19
Updated: 2026-11-16
Code: https://github.com/cykbennie/fbi
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 90/100
The gist: We develop tests of clustered equal predictive ability (C-EPA) in panel data where clusters are unknown and must be estimated from the data using the Panel Kmeans algorithm.
Key concepts
- Panel Kmeans
- This algorithm is used to estimate latent clusters in panel data. It groups observations based on their similarities across different time periods or entities in the dataset, effectively revealing hidden structures without prior knowledge of how many clusters exist.
- Selective Conditional Inference Framework
- This method addresses the problem that tests are based on clusters estimated from the data itself. Instead of relying on sample splitting, this framework uses a procedure that conditions inference on the estimated cluster structure to ensure valid results.
- Truncated $\chi$ Variable
- The limiting distribution of a specific test statistic is shown to be that of a truncated chi-squared ($\chi$) variable. This mathematical property is crucial because it allows for accurate size control and understanding of the test's behavior under the estimated cluster structure.
- $FSI_r$ Statistic
- This is the main test statistic proposed by the authors. It combines tests for pairwise equality of cluster centers with a test for overall Equal Predictive Ability (EPA). It provides a valid p-value even when multiple hypotheses are tested simultaneously, using a generalized mean of order r.
Terminology
Summary
We develop tests of clustered equal predictive ability (C-EPA) in panel data where clusters are unknown and must be estimated from the data using the Panel Kmeans algorithm. p. 0 Testing Clustered Equal Predictive Ability with Unknown
Clusters∗
O˘guzhan Akg¨un† Alain Pirotte‡ Giovanni Urga§ Zhenlin Yang¶
July 10, 2026
Abstract
We develop tests of clustered equal predictive ability (C-EPA) in panels where the clusters are
unknown and estimated by the Panel Kmeans algorithm. To address the challenge of testing
hypotheses that depend on data-driven clusters, we adopt a selective conditional inference
framework.Where the paper says it
double dipping
problem inherent when clustering is used in both hypothesis testing and cluster estimation.
The core methodology involves deriving a Wald-type test for pairwise equality of cluster centers, demonstrating that its limiting distribution, conditional on the estimated clusters, is that of a truncated chi variable. p. 0 To address the challenge of testing
hypotheses that depend on data-driven clusters, we adopt a selective conditional inference
framework. Specifically, we first derive a Wald-type test for pairwise equality and show that
the limiting distribution of its square root conditional on the estimated clusters is that of a
truncated χ variable. We characterize the associated truncation set by quadratic inequalities
in the data space. Then, for the C-EPA hypothesis, we propose a p-value combination method
by aggregating the evidence against the pairwise equality and overall EPA null hypotheses. Notation Random variables are denoted by upper-case letters and their realizations by the
corresponding lower-case letters, e. , w(·) denotes a realization of the test statistic W(·). Then, for the C-EPA hypothesis, we propose a p-value combination method
by aggregating the evidence against the pairwise equality and overall EPA null hypotheses. The Monte Carlo results show accurate size control and good finite-sample power of the
proposed tests.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The central inferential difficulty lies in the fact that the clusters used in the C-EPA null are estimated from loss differentials on which the test itself is based, creating a post-selection problem. p. 2 We also show that selecting
the number of clusters by an information criterion does not invalidate the proposed procedure,
and we examine the role of multiple Kmeans initializations in simulations. The central inferential difficulty is that the clusters used in the C-EPA null are estimated
from the same loss differentials on which the test is based. , 2022; Patton and Weller, 2023). However, once
clusters are data-driven, conventional tests that treat them as fixed suffer from a post-selection
problem: the clustering step mechanically separates units in ways that can create spurious
evidence of heterogeneity. The split between training and testing samples is arbitrary,
the testing sample may be substantially smaller, and the procedure can be sensitive to structural
breaks or other forms of temporal instability. These limitations motivate our selective conditional
approach, which uses the full sample while conditioning on the clustering outcome.Where the paper says it
Where the paper says it
Where the paper says it
The framework advances four main directions: first, formulating C-EPA testing in a conditional predictive ability setting; second, estimating latent clusters via Panel Kmeans and showing its equivalence to standard Kmeans applied to unit-level time averages; third, developing the selective conditional inference procedure that yields valid post-clustering inference (specifically, the square root of a Wald-type statistic has an asymptotic truncated chi distribution conditional on the selected clustering outcome); and fourth, constructing tests of C-EPA when clusters are unknown. p. 1 Second, we estimate
latent clusters using Panel Kmeans and show that, for our purpose, Panel Kmeans is equivalent
to standard Kmeans applied to unit-level time averages, thereby reducing the selective-inference
problem from NT P to NP dimensions. Third, we develop a selective conditional inference
procedure that delivers valid post-clustering inference: for pairwise equality of cluster centers,
the square root of a Wald-type statistic has an asymptotic truncated χ distribution conditional
on the selected clustering outcome.Where the paper says it
The main result is Theorem 7, which shows that the proposed FSI r statistic controls the Type I error rate asymptotically and remains consistent even if either the homogeneity null (H homo) or the overall EPA null (H epa) fails. p. 8 For simplicity, here and throughout we suppress the dependence of p[ · ] on
the reference distribution, which will be clear from the context as we establish the asymptotic
distribution of each test, under the respective null. A level-α test rejects the null if p[w(C)] ≤ α,
where α ∈ (0, 1) is the predetermined Type I error rate. The proposed procedures are evaluated in Monte Carlo simulations and compared with
naive post-clustering tests and split-sample alternatives. The simulations show that naive tests
severely over-reject, while the selective tests control size accurately and retain good power in
empirically relevant designs. dollar, we compare
traditional time-series models and machine-learning methods relative to an AR(1) benchmark. The results show that accounting for latent clusters can reveal heterogeneous predictive gains
that are missed by aggregate forecast-comparison tests.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop a selective conditional inference procedure for testing clustered equal predictive ability when clusters are unknown by using Panel Kmeans to estimate latent clusters and conditioning on this estimated clustering outcome to control Type I error rates.
-
Implement a p-value combination method by aggregating evidence against the pairwise equality null hypotheses and the overall equal predictive ability null hypothesis, utilizing a generalized mean of order r defined as Fr for arbitrary dependence among p-values.
-
Create an information criterion for estimating the number of clusters, IC(K), to replace Bonferroni correction, which is computationally more efficient than cross-validation and avoids conditioning on the choice of the information criterion itself.
-
Design a test statistic FSI,r that combines pairwise equality tests with an overall equal predictive ability test, providing a valid p-value under arbitrary dependence among the input p-values by selecting a value of r within the admissible range [-infinity, -1].
-
Enable accurate size control and good finite-sample power for C-EPA tests by replacing naive post-clustering tests with selective inference procedures that account for the data-dependent nature of cluster formation via truncation based on the estimated clusters.
Abstract
We develop tests of clustered equal predictive ability (C-EPA) in panels where the clusters are unknown and estimated by Panel Kmeans. Since the hypotheses depend on data-driven clusters, we adopt a selective conditional inference framework. We show that the square root of a Wald statistic for the equality of two estimated cluster centers is asymptotically a truncated χ variable, conditionally on the selected clusters and their number, with multiple random initializations, and that the truncation set is a finite union of intervals that can be computed exactly. Unlike existing methods of selective inference for Kmeans, this holds under serial and cross-sectional dependence and heteroskedasticity. For the C-EPA hypothesis, we combine the pairwise p-values with a test of overall EPA. In Monte Carlo experiments, the proposed tests control size where naive tests severely over-reject. An application to exchange rate forecasting with time series and machine learning methods illustrates the procedure.
Sources
- Consistent Selection of the Number of Groups in Panel Models via Cross-Validation
- Unifying approach to selective inference with applications to cross-validation
Related papers
- Stationarity with Occasionally Binding Constraints
- Backtesting High-Dimensional Expected Shortfall
- Higher-Order Neyman Orthogonality in Moment-Condition Models
- Online Bayesian Inference for a Dirichlet Process Mixture of Binary Logit Models
- Debiased Conditional Independence Tests for Causal Effects
- Sharp Testable Implications of Encouragement Designs