Sequential Conditional Independence Testing with Machine Learning Models

The gist Sequential Conditional Independence Testing with Machine Learning Models explains how to incorporate machine learning models into sequential conditional independence testing by decomposing error and exploring intermediate null hypotheses to achieve robust and powerful e-variables.

Model-X Assumption and Null Hypotheses

The model-X assumption shifts the burden of modeling toward the dependence structure among the inputs themselves, relying on the idea that relationships among inputs are often easier to model than their relationship with the output. The paper introduces several null hypotheses in a hierarchy, including Px-CI K for Model-X Conditional Independence and Pexch B for B-Exchangeability. These intermediate nulls allow for studying when tests are safe and which e-variable is the Growth Rate Optimal (GRO) e-variable and their corresponding GRO e-variables.

E-Variable Classes and Optimality

The paper compares several classes of e-variables to test these nulls, including the GRO e-value against a point alternative Q and the antisymmetric e-value. The exponential e-variable is proposed by constructing pseudoprobabilities by exponentiating the loss with a variance-like parameter η and the coin-betting e-value is an instantiation of the antisymmetric e-value where there is no need to apply the sign function. The analysis shows that neither method uniformly dominates the other, as they introduce different types of errors such as null enlargement error and approximation error.

Analysis under the Alternative and Power Growth

The analysis decomposes the e-power loss into null enlargement error, approximation error, and estimation error. The paper derives bounds on the type-I error inflation when the model-X distribution is estimated using triple robustness results. In the small-signal regime, the exponential e-power grows quadratically with the signal strength ∆, while coin-betting e-power also grows quadratically with ∆ and.

Sequential Estimation and Double Robustness

The algorithms presented are agnostic to how the model is retrained, accommodating any machine learning model and work for any classification or regression task for which a loss measuring performance can be provided>. The analysis accounts for the estimation of the conditional distribution by deriving less conservative bounds on the type-I error inflation using double robustness results and showing that even with estimated kernels, the statistic becomes increasingly insensitive to discrepancies in the X-distribution as the fitted model approaches the null regression function.

Practical Implications and Safety Guarantees

The paper demonstrates that none of the e-processes reject for the null mutation, showing robustness to violations of the model-X assumption and that they converge to true quantities under certain conditions. The practical application suggests that mixture of tests using previous data points can be data-driven and dynamic over time, reducing the null enlargement error and that the GRO property of Eexch holds at B = 1 but is not preserved after derandomization.

Conclusion on Testing Strategies

The study concludes that the sequential world allows for successively selecting a mixture of tests using the previous data points, where approximation/null enlargement weighting can be data-driven and dynamic over time and that incorporating model training into e-variables provides a more reliable tool for sequential testing using ML.

--- Page 40 ---

The gist Sequential Conditional Independence Testing with Machine Learning Models explains how to incorporate machine learning models into sequential conditional independence testing by decomposing error and exploring intermediate null hypotheses to achieve robust and powerful e-variables.

How it works

The paper decomposes the error into null enlargement, approximation, and estimation error. It explores intermediate null hypotheses between model-X conditional independence and exchangeability to reduce these errors and provides estimation error bounds that accommodate triple robustness results achieving fast convergence rates.

Analysis under the Alternative and Power Growth

The analysis decomposes the e-power loss into null enlargement error, approximation error, and estimation error. The paper derives bounds on the type-I error inflation when the model-X distribution is estimated using triple robustness results achieving fast convergence rates. In the small-signal regime, the exponential e-power grows quadratically with the signal strength ∆, while coin-betting e-power also grows quadratically with ∆ and.

1. AI systems can perform sequential conditional independence testing by leveraging e-values, allowing for continuous monitoring of data and flexible inference procedures to decide when additional data collection is necessary or sufficient evidence has been obtained.

2. The AI system can utilize a Sequential Conditional Independence Testing with Machine Learning Models approach to sequentially assign the importance of a feature within a black-box machine learning model under the model-X assumption by comparing the loss of the original model to that of a modified version where the feature’s information is removed and replaced with a conditionally independent counterpart.

3. The system can employ an exponential e-value or coin-betting e-value to sequentially estimate evidence, with coin-betting potentially outperforming the exponential setting in certain small-signal regimes by optimizing betting parameters like λ.

4. The AI system can implement an antisymmetric e-value using importance statistics (e.g., the loss of a model) and test their exchangeability under a larger null hypothesis, which can yield higher e-power than the original model-X conditional independence null by testing exchangeability or symmetry.

5. The system can use intermediate null hypotheses derived from progressive information distillation, such as the PFX-CI null (testing conditional independence based on importance statistics), to generate e-values with a lower null enlargement error compared to the original model-X null hypothesis.

6. The AI system can implement double robustness by using loss functions based on both the estimated conditional distribution and the fitted model, ensuring that even if the estimated kernel is imperfect, the test statistic remains insensitive to discrepancies between distributions as long as certain convergence rates hold.

Conditional independence testing is a ubiquitous problem in scientific discovery. The widely employed model-X assumption shifts the modelling burden from the dependence of the output on the inputs to the dependencies within the inputs. Log-optimal e-variables have been studied in this setting, but it remains unclear how to incorporate machine learning models into their design. Other approaches test exchangeability directly, yielding an e-variable with lower power in theory but, surprisingly, higher power in practice. We explain this phenomenon by decomposing the error into null enlargement, approximation, and estimation error. The decomposition shows that GRO e-variable estimates can be beaten because of their worse approximation and estimation errors, and we explore intermediate null hypotheses between model-X conditional independence and exchangeability to reduce these errors. Moreover, the model-X assumption often only holds up to an estimation error, invalidating exact type-I error guarantees. We provide estimation error bounds that accommodate triple robustness results, achieving fast convergence rates.