Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models
The paper investigates how a foundation model's ability to handle out-of-distribution (OOD) data is determined by the spectral structure of its pretraining weights. It shows that this structure is shaped by the combination of model architecture and pretraining strategy, creating a unique 'spectral geometry.' This geometry governs OOD robustness, allowing researchers to diagnose potential weaknesses using only released weights before any target data is used.