Revisiting Generalization Measures Beyond IID: An Empirical Study under Distributional Shift
Sora Nakai, Youssef Fadhloun, Kacem Mathlouthi, Kotaro Yoshida, Ganesh Talluri, Ioannis Mitliagkas, Hiroki Naganuma
cs.LG
Submitted: 2026-08-21
Updated: 2026-08-24
Comments: Accepted to TMLR
License: http://creativecommons.org/licenses/by/4.0/
The gist: Predicting generalization from quantities available before target-test evaluation remains a central challenge in deep learning.
Terminology
Abstract
Predicting generalization from quantities available before target-test evaluation remains a central challenge in deep learning. The systematic benchmark of Jiang et al. (2020) evaluated many generalization measures, but it focused on independent and identically distributed (IID) settings. We revisit this problem for image classifiers evaluated under controlled corruptions and perturbations. Our study uses CIFAR-10-C/P, where the label space and task remain fixed while the input images are degraded or perturbed. This setting also allows us to revisit the robustness concerns raised by Dziugaite et al. (2020), who showed that the apparent reliability of generalization measures can depend strongly on experimental conditions. Our experiments show that the usefulness of generalization measures is strongly regime-dependent. In our exploratory decision analysis across three CNN-style architectures, sharpness- and input-gradient-based measures are among the leading individual signals, whereas family results are close and architecture dependent. Optimization-based measures, Information Criteria, and Sharpness-based measures provide additional regime-dependent signals in correlation or local-reliability analyses. Together, these findings suggest that model selection should not rely only on measures favored by IID evaluation. Instead, within the evaluated CIFAR-10-C/P setting and architectures, generalization measures should be treated as regime-dependent ranking signals whose utility must be evaluated for the intended corruption or perturbation setting.
Sources
- Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
- Invariant Risk Minimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Averaging Weights Leads to Wider Optima and Better Generalization
- Sharpness-Aware Minimization for Efficiently Improving Generalization
- Measuring Robustness to Natural Distribution Shifts in Image Classification
- Network In Network
- An Empirical Model of Large-Batch Training
- In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
- Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks