CORE-STACK+: Meta-Learning for Deep Stacked Generalization
cs.LG, cs.CV
Submitted: 2026-09-22
Updated: 2026-09-22
License: http://creativecommons.org/licenses/by/4.0/
The gist: Stacking heterogeneous vision backbones (CNNs, ViTs, and hybrids) is the de facto recipe for accuracy, calibration, and robustness, yet two coupled pathologies limit its returns.
Terminology
Abstract
Stacking heterogeneous vision backbones (CNNs, ViTs, and hybrids) is the de facto recipe for accuracy, calibration, and robustness, yet two coupled pathologies limit its returns. Prediction-space multicollinearity ill-conditions the meta-learner's Gram matrix, inflating weight variance and producing brittle solutions on a thin manifold. Calibration collapse compounds constituent miscalibration through naive linear stacking, so adding more models can hurt expected calibration error (ECE). Existing remedies, ridge regularization, greedy selection, model soups, and SWAG address at most one of these issues, and none jointly target conditioning and calibration in heterogeneous prediction pools. We introduce CORE-STACK+, a preconditioning pipeline with four components: (i) a kernelized redundancy filter that removes non-linear inter-model dependencies invisible to Pearson correlation, using Centered Kernel Alignment (CKA) [23]; (ii) a <15 K-parameter differentiable meta-feature gate that learns per-sample attention over ensemble statistics; (iii) a spectrum-adaptive Ridge penalty lambda star=lmax(Chat)/SNR(Chat) derived from a Marchenko-Pastur signal-noise decomposition, eliminating nested cross-validation; and (iv) a Laplace-approximate Bayesian blender replacing inverse-RMSE heuristics. We prove a PAC-Bayes excess-risk bound that, for the first time, jointly accounts for prediction-space redundancy and meta-learner capacity. Across six benchmarks, CORE-STACK+ delivers +1.8% top-1 on ImageNet-1K, -4.2 mCE on ImageNet-C, +0.9 mIoU on ADE20K, and +1.3 AP on COCO, while reducing retained models by 35-57% and inference FLOPs by up to 41%. ECE improves 2.1 times over deep ensembles without post hoc temperature scaling.
Sources
- Repulsive Deep Ensembles are Bayesian
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
- Deep Ensembles: A Loss Landscape Perspective
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- On Calibration of Modern Neural Networks
- Mask R-CNN
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Natural Adversarial Examples
- Snapshot Ensembles: Train 1, get M for free
- Averaging Weights Leads to Wider Optima and Better Generalization
- Similarity of Neural Network Representations Revisited
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Microsoft COCO: Common Objects in Context
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- A ConvNet for the 2020s
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Learning under Model Misspecification: Applications to Variational and Ensemble methods
- Second Order PAC-Bayesian Bounds for the Weighted Majority Vote
- A Note on the PAC Bayesian Theorem
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks