The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
The paper critiques standard statistical methods for estimating coverage guarantees in machine learning thresholds when data is clustered. It introduces the Exceedance Design Effect, showing that a single effective sample size is insufficient. Instead, the required sample size depends on the specific threshold level chosen and how correlated data points cluster.