On the Invariance and Generality of Neural Scaling Laws
The paper develops a framework using information theory to understand how neural scaling laws change when data is transformed. It distinguishes between data transformations that preserve information and those that degrade it, quantifying this degradation through variance inflation and optimal loss shifts. This allows for predicting model performance in new domains by estimating the 'information resolution' of the data.