Optimal Inference with Black-box Predictions

This paper studies how to optimally combine observed data with black-box predictions to perform valid and efficient statistical hypothesis testing in the high-dimensional Gaussian sequence model. The authors characterize the information-theoretic limits of inference with black-box predictions when their accuracies are known and, for orthogonal predictions, when they are unknown. They develop practical hypothesis tests that adapt to the unknown accuracies of the predictions while benefiting from strong alignment among them.

The paper considers the setting where one has access to n observations from a standard d-dimensional Gaussian distribution X1,..., Xn ∼ N(θ, Id) with unknown mean θ ∈ Rd, and m predictions of the mean θ̂1,..., θ̂m whose accuracy is unknown. The goal is to detect whether the mean of the data is zero (θ = 0) or far from it.

The main results are:

• They characterize the optimal testing strategy of an oracle test that has access to the unknown predictions' accuracies (Theorems 1 and 2).

• For orthogonal predictions, they develop a test that adapts to the unknown accuracy of the predictions and establish its optimality (Theorems 3 and 4).

• For arbitrary predictions, they develop a test that adapts to the unknown accuracy of the predictions while benefiting from high cosine similarity among the predictions (Theorem 5).

The paper establishes that without predictions, the hypotheses can be distinguished with nontrivial power if and only if n ≳ √d · ϵ−2, which is called the prediction-free rate. The optimal test is based on the chi-squared test statistic.

The authors show that arbitrary predictions can substantially improve the prediction-free rate without requiring strong assumptions about their quality. Informative predictions help the test focus on a smaller set of directions where the null distribution may fail, which increases power when the predictions are reliable. Poor predictions, however, can point the test in unhelpful directions and reduce its effectiveness. They address this tension with tests that adapt to the unknown accuracy of the predictions, which they call adaptive tests.

For the oracle setting where the accuracy of a single prediction is known, the paper characterizes the optimal performance. The key quantity governing the interpolation between extreme cases is γ∗(ϵ, r), the length of the smallest projection onto the span of the prediction under the alternative hypothesis. Theorem 1 states that if the worst cosine similarity between the mean of the data distribution and its projection onto the prediction is low, then the prediction is not informative enough, and the prediction-free rate is obtained. However, if the cosine similarity is large enough, the oracle can project the data onto the prediction and make a decision based on this one-dimensional quantity, avoiding any dependence on the ambient dimension of the data.

For multiple predictions, the oracle test can aggregate them optimally and perform a one-dimensional test by combining their projections. Theorem 2 shows that when predictions are informative, the performance of the oracle test is independent of the dimension of the data.

The main focus is on adaptive tests that do not know the accuracy of the predictions. For orthogonal predictions, the adaptive test monitors for deviations from the null distribution in all predicted directions. Theorem 3 shows that the adaptive test can require more observations than the oracle test, with the ratio between these sample sizes, called the adaptivity gap, being as large as √m. This gap arises because the adaptive test must search for deviations throughout the span of the predictions, whereas the oracle test uses the predictions' accuracies to aggregate the predicted directions into a single direction.

Theorem 4 establishes that this dependence on the number of predictions is optimal among separation curves satisfying a natural invariance property, which states that accuracy vectors that lead to equally hard worst-case distributions under the alternative hypotheses get assigned the same separation.

For arbitrary predictions, the paper introduces a test based on the intrinsic projection of the data onto the span of the predictions, which automatically downweights directions that align poorly with the predictions. The performance of this test depends on the intrinsic dimension of the squared cosine similarity matrix, Int Σ2 = Tr Σ2 /∥Σ2∥op, rather than the ambient dimension of the span of the predictions. Theorem 5 shows that when the predictions share substantial information, the adaptive test depends on the lower intrinsic dimension rather than on the full ambient dimension of the predictions' span.

The paper also includes simulations that illustrate the performance of the proposed tests. The simulations show that the adaptive test achieves a rate-wise free lunch: it closely tracks the power of the projection test when the prediction is good but remains close to the power of the chi-squared test when the prediction is uninformative. The simulations also demonstrate that tests based on intrinsic projection benefit from alignment among predictions and outperform those based on standard projection when there is high cosine similarity among the predictions.

The paper concludes by discussing the implications of the results, noting that testing with black-box predictions achieves faster rates than comparable estimation problems with similar guarantees. It also highlights that testing requires only information about the direction of a prediction, suggesting a practical criterion for training predictive models when their outputs feed into downstream decisions.

Improvements to AI Systems:

1. Adaptive Hypothesis Testing with Black-Box Predictions: AI systems can now perform statistical hypothesis tests that automatically adapt to the unknown accuracy of external predictive models (e.g., neural networks, ensemble methods) without requiring retraining or calibration. This enables robust decision-making (e.g., anomaly detection, A/B testing, quality control) even when the predictions are unreliable, by dynamically switching between a chi-squared test (prediction-free) and a projection-based test based on the data's alignment with predictions.

2. Dimension-Reduced Inference via Intrinsic Projection: The system can leverage the intrinsic dimension of the prediction space (computed from the squared cosine similarity matrix) to reduce the effective dimensionality of the test. This allows AI systems to handle high-dimensional data (e.g., genomics, imaging) with many correlated predictions, achieving faster detection rates (sample complexity proportional to intrinsic dimension rather than ambient dimension) while automatically downweighting uninformative prediction directions.

3. Oracle-Like Performance without Accuracy Knowledge: The system can achieve near-optimal power (matching an oracle that knows prediction accuracies) for orthogonal predictions, with a provable adaptivity gap of at most √m (where m is the number of predictions). This means AI systems can combine multiple black-box models (e.g., different pre-trained models) without knowing their individual performance, and still maintain statistical validity and efficiency.

4. Robustness to Poor Predictions: The system can automatically detect when predictions are uninformative (low cosine similarity with the true signal) and fall back to the prediction-free chi-squared test, preventing loss of power. This is critical for AI systems that rely on external predictions (e.g., from foundation models) that may be misaligned with the task.

5. Practical Training Criterion for Predictive Models: The paper’s insight that testing only requires direction (not magnitude) of predictions suggests a new training objective: optimize models to produce predictions with high cosine similarity to the true signal, rather than minimizing squared error. AI systems can use this to train models that are specifically designed to feed into downstream hypothesis tests, improving downstream decision accuracy with fewer samples.

6. Scalable Multi-Prediction Aggregation: The system can aggregate multiple predictions (even non-orthogonal ones) into a single test statistic via intrinsic projection, benefiting from shared information among predictions. This enables AI systems to fuse outputs from diverse models (e.g., vision, language, tabular) for hypothesis testing, with performance improving as predictions align, without needing to know their accuracies a priori.

What the Improved AI System Can Do:

• Detect subtle signals in high-dimensional data (e.g., rare disease markers, network intrusions) using any number of black-box predictions, with sample complexity that adapts to prediction quality and intrinsic dimensionality.

• Provide statistically valid p-values and confidence bounds even when predictions are adversarial or noisy, ensuring reliable decision-making in safety-critical applications.

• Automatically choose the optimal test strategy (projection vs. chi-squared) based on data, without manual tuning or prior knowledge of model performance.

• Train new predictive models that are optimized for downstream hypothesis testing, reducing the data required for detection by focusing on directional accuracy.

Powerful black-box predictive models have motivated many proposals for combining observed data with predictions to perform valid statistical inference. Despite this progress, the field lacks a unifying principle that explains how hypothesis tests should integrate data and predictions in a way that is both valid and efficient. In this work, we address this gap in the high-dimensional Gaussian sequence model. We characterize the information-theoretic limits of inference with black-box predictions when their accuracies are known and, for orthogonal predictions, when they are unknown. Building on these characterizations, we develop practical hypothesis tests that adapt to the unknown accuracies of the predictions while benefiting from strong alignment among them.