Testing Clustered Equal Predictive Ability with Unknown Clusters
summary
The gist
We develop tests of clustered equal predictive ability (C-EPA) in panel data where clusters are unknown and must be estimated from the data using the Panel Kmeans algorithm.
In short
The study develops a test for clustered equal predictive ability (C-EPA) when clusters are unknown, estimated using Panel Kmeans. It proposes a selective conditional inference framework to avoid 'double dipping' when clustering is used for both testing and estimation. The main result is a robust test statistic that controls Type I errors and maintains power, proving that conditioning on estimated clusters provides valid post-clustering inference.
Key concepts
- Panel Kmeans
- This algorithm is used to estimate latent clusters in panel data. It groups observations based on their similarities across different time periods or entities in the dataset, effectively revealing hidden structures without prior knowledge of how many clusters exist.
- Selective Conditional Inference Framework
- This method addresses the problem that tests are based on clusters estimated from the data itself. Instead of relying on sample splitting, this framework uses a procedure that conditions inference on the estimated cluster structure to ensure valid results.
- Truncated $\chi$ Variable
- The limiting distribution of a specific test statistic is shown to be that of a truncated chi-squared ($\chi$) variable. This mathematical property is crucial because it allows for accurate size control and understanding of the test's behavior under the estimated cluster structure.
- $FSI_r$ Statistic
- This is the main test statistic proposed by the authors. It combines tests for pairwise equality of cluster centers with a test for overall Equal Predictive Ability (EPA). It provides a valid p-value even when multiple hypotheses are tested simultaneously, using a generalized mean of order r.
Terminology used across episodes
This episode discusses
- Testing Clustered Equal Predictive Ability with Unknown Clusters · Paper Radio
- Consistent Selection of the Number of Groups in Panel Models via Cross-Validation
- Unifying approach to selective inference with applications to cross-validation
The paper
Testing Clustered Equal Predictive Ability with Unknown Clusters · Read on arXiv
University of Bourgogne Europe · Paris-Panthພon-Assas University · Bayes Business School (formerly Cass) · School of Economics, Singapore Management University
We develop tests of clustered equal predictive ability (C-EPA) in panels where the clusters are unknown and estimated by Panel Kmeans. Since the hypotheses depend on data-driven clusters, we adopt a selective conditional inference framework. We show that the square root of a Wald statistic for the equality of two estimated cluster centers is asymptotically a truncated χ variable, conditionally on the selected clusters and their number, with multiple random initializations, and that the truncation set is a finite union of intervals that can be computed exactly. Unlike existing methods of selective inference for Kmeans, this holds under serial and cross-sectional dependence and heteroskedasticity. For the C-EPA hypothesis, we combine the pairwise p-values with a test of overall EPA. In Monte Carlo experiments, the proposed tests control size where naive tests severely over-reject. An application to exchange rate forecasting with time series and machine learning methods illustrates the procedure.
Transcript
Introduction to the show: ident: Economics. The day in economics: theory, mechanisms and the evidence beneath them.
Ruth: We're looking at the paper: "Testing Clustered Equal Predictive Ability with Unknown Clusters".
Warren: Here's the gist of the paper. We develop tests of clustered equal predictive ability (C-EPA) in panel data where clusters are unknown and must be estimated from the data using the Panel Kmeans algorithm.
Ruth: To address the challenge of testing hypotheses dependent on these data-driven clusters, we propose a selective conditional inference framework designed to mitigate the "double dipping" problem inherent when clustering is used in both hypothesis testing and cluster estimation.
Warren: The core methodology involves deriving a Wald-type test for pairwise equality of cluster centers, demonstrating that its limiting distribution, conditional on the estimated clusters, is that of a truncated chi variable.
Ruth: We then propose a p-value combination method by aggregating evidence against both the pairwise equality null hypotheses and the overall Equal Predictive Ability (EPA) null hypothesis.
Title and authors: Ruth: To wrap up what we just heard about "Testing Clustered Equal Predictive Ability with Unknown Clusters," the paper essentially introduces a method for testing equal predictive ability in panel data when you don't know the clusters that generate that predictive ability.
Warren: They use Panel Kmeans to estimate these unknown clusters, and then they propose a selective conditional inference framework to test hypotheses about clustered equal predictive ability by conditioning on those estimated clusters.
Iris: It’s addressing the problem of double dipping where you use data-driven clusters both for estimation and testing, which is a known issue in this literature.
Ruth: Specifically, they derive a Wald-type test for pairwise equality and show that its square root conditional on the estimated clusters follows a truncated chi variable distribution.
Warren: And then for the overall equal predictive ability hypothesis, they use a p-value combination method by aggregating evidence from both the pairwise equality tests and the overall EPA null hypothesis.
Iris: The main statistic they derive is FSI r, which allows them to combine these pieces of evidence under some arbitrary dependence among the input p-values by selecting a value of r within a specific range.
Ruth: The authors are also looking at how to select the number of clusters using an information criterion, IC(K), instead of relying on something like Bonferroni correction, which they say is computationally more efficient and avoids conditioning on the choice of the criterion itself.
Warren: And they show that using multiple Kmeans initializations in their simulations doesn't invalidate their procedure, so they’re checking for robustness there as well.
Iris: The paper also examines the role of multiple Kmeans initializations in simulations to see if the result holds up under different starting points for the clustering algorithm.
The paper's summary: Ruth: So, summarizing what they actually did, this paper develops tests of clustered equal predictive ability when clusters are unknown and must be estimated using a method like Panel Kmeans.
Warren: The core contribution is moving beyond the assumption that we know the clusters beforehand by proposing a selective conditional inference framework to handle these data-driven clusters.
Iris: They first build a Wald-type test for pairwise equality and show its limiting distribution when conditioned on the estimated clusters is that of a truncated chi variable, which helps control Type one errors.
Ruth: Then, they propose combining this with an overall equal predictive ability test using a p-value merging function valid under dependence among the components to get their main statistic FSI r.
Warren: The authors also show that selecting the number of clusters via an information criterion, IC(K), is fine and doesn't break the proposed procedure.
Iris: The central inferential difficulty they address is that conventional tests treat these data-driven clusters as fixed, which causes spurious evidence of heterogeneity due to this post-selection problem.
Ruth: Their solution is to use the full sample while conditioning on the clustering outcome, which avoids relying on sample splitting or external cluster information.
Warren: The paper confirms their method works well in simulations, showing accurate size control and good finite-sample power for both the pairwise equality tests and the overall EPA test.
Iris: Their empirical application to exchange-rate forecasting shows that this procedure is practically relevant because it reveals heterogeneous predictive gains missed by aggregate tests when latent clusters are accounted for.
The paper's improvements: Ruth: One of the main improvements they suggest is replacing naive post-clustering tests with their selective inference procedures that account for the data-dependent nature of cluster formation via truncation based on the estimated clusters.
Warren: That’s a direct fix for the double dipping issue, and it moves away from relying on external information or sample splitting to control Type one error rates.
Iris: They also developed a test statistic FSI,r that combines pairwise equality tests with an overall equal predictive ability test, providing a valid p-value under arbitrary dependence among the input p-values by selecting a value of r within the admissible range of negative values.
Ruth: And they introduced the information criterion for estimating the number of clusters, IC(K), which is computationally cheaper than cross-validation and avoids conditioning on which specific criterion you choose.
Warren: That's a practical improvement because it makes choosing how many clusters to estimate less dependent on an arbitrary choice of an external selection tool.
Iris: They also showed that the square root of the Wald-type statistic has an asymptotic truncated chi distribution conditional on the selected clustering outcome, which is a very specific distributional result they want to prove.
Ruth: So, these improvements are all aimed at making sure that this test isn't just theoretically sound but also practically usable for controlling size and maintaining power in real applications.
Conclusion: Warren: So to wrap up the "Testing Clustered Equal Predictive Ability with Unknown Clusters," the paper proves that their selective conditional inference test, FSI r, controls the type one error rate asymptotically and remains consistent even when either the homogeneity null or the overall equal predictive ability null fails.
Iris: That means this framework is a robust way to handle unknown clusters without needing prior knowledge of them, because it accounts for model selection uncertainty internally.
Ruth: The final conclusion is that this selective conditional inference test offers a method that controls size effectively and is preferred over naive post-clustering tests because it directly addresses how data-driven cluster formation affects the inference.
Warren: They confirmed this with empirical work in exchange-rate forecasting, showing that accounting for latent clusters shows heterogeneous predictive gains missed by aggregate tests, which really validates the approach.
Iris: I think what this means for us is that when we look at panel data, we should stop looking for simple aggregate effects and start conditioning on the cluster structure estimated from the data itself if those structures matter.
More episodes
- Daily Summary for 2026-11-19
- 2608.21274-Recommendation Quality and the Concentration of Consumption: Experimental Evidence from Netflix
- 2408.02757-A nonparametric test for diurnal variation in spot correlation processes
- 2306.16393-High-Dimensional Canonical Correlation Analysis
- 2605.17299-Geometric Brownian motion with intermittent entries and exits
- 2607.04753-Fooling Yourself: how narratives shape beliefs
- 2408.14671-Inference on Treatment Effects with Repeatedly Mismeasured High-Dimensional Controls
- 2201.06898-Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period
- 2501.15307-Influence Functions: A Unified Differentiation-Based Framework
- The day in conversation for 2026-11-19