From Objectives to What Models Learn: A Landau Theory of Invariant Learning

The paper From Objectives to What Models Learn: A Landau Theory of Invariant Learning by Pinli Wang, Yue He, and Peng Cui addresses the objective–behavior gap in invariant learning, where the behavior of objectives along the regularization path is often opaque. The authors propose a Landau-type effective free energy derived from concrete invariant-learning objectives to predict regularization phenotypes.

Abstract Summary:

The paper states: "Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective–behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes. The authors find that Effective quadratic corrections move the phase boundary and enable finite-strength mode elimination; quartic corrections regulate post-onset amplitude and typically leave residual loading at finite strength; higher-order structure governs non-monotone tails, instability, and collapse at large regularization. In a canonical bilinear model, the theory yields closed-form phase boundaries and steady-state loadings, as well as distinct critical strengths for shortcut and stable modes that define a selective-retention window." The framework is validated in controlled experiments and extended to ReLU networks and matrix extensions for coupled collective modes.

Introduction Summary:

The paper notes that objectives built around this common principle can behave very differently, while their formulas provide little indication of what will happen along the regularization path. The authors view predictive-mode learning through the lens of magnetization, where A mode’s learned loading serves as the order parameter, regularization strength as a temperature-like control, and environment-dependent support as a mode-dependent external field. The contributions are: (i) Ferromagnetic phase-transition perspective, (ii) Objective signatures as a predictive methodology, (iii) Structural analysis of existing objectives, and (iv) Empirical validation.

Formulation Summary:

The paper considers supervised learning across training environments e ∈ Etr with joint distributions Pe (X, Y). A stable feature remains predictive across environments, whereas a shortcut is predictive only under particular training conditions. The authors study equilibria of the learning objective, namely the stable states reached at the end of optimization. Using the bilinear parameterization β = wθ, the objective obeys F(−w, −θ; λ) = F(w, θ; λ), which is the Z2 symmetry of a zero-field Landau free energy. The effective free energy is expanded as Feff (w; λ) = L0 + (1/2)r(λ)w2 + (1/4)u(λ)w4 + O(w6), where r is the actual quadratic curvature and u collects fourth-order contributions. The canonical objective is Fcan (w, θ; λ2, λ4) = L0 − awθ + (m + λ4 γ2)/2 w2θ2 + (µ + λ2 γ2)/2 w2 + µ/2 θ2, where "a is the task-predictive signal, m is the local task-loss curvature with respect to the final loading β, γ ≥ 0 is the environmental sensitivity, and µ > 0 is the baseline quadratic curvature."

Single Feature Theory Summary:

For a single feature, the reduced objective is Feff (w) = L0 + (A/2)w2 − a2w2/(2(µ + Bw2)), with A = µ + λ2 γ and B = m + λ4 γ. The Landau expansion gives Feff (w) = L0 + (γ/2)(λ2 − λ2,c)w2 + (u/4)w4 + O(w6), with λ2,c = (a2/µ − µ)/γ and u = 2a2B/µ2 > 0. The paper states: λ2 is a temperature-like control parameter and w is the order-parameter amplitude of the learning system. Theorem 4.1 gives the phase boundary: "a2 > µ(µ + λ2 γ) ⇐⇒ λ2 < λ2,c, λ2,c = (a2/µ − µ)/γ. Corollary 4.2 states: If λ2,c(s) < λ2,c(c), then for every λ2 ∈ (λ2,c(s), λ2,c(c)), the shortcut is stabilized at the origin while the stable direction remains in the nonzero acquired phase. Proposition 4.3 gives the steady-state loading: β ⋆ = [a − √(µ(µ + λ2 γ))]+ / (m + λ4 γ). Thus, R2 makes the loading vanish exactly at the finite boundary λ2,c, whereas R4 continuously shrinks its nonzero post-onset loading."

Multi-Component Order Parameters Summary:

The paper extends to multicomponent order parameters, where distinct components can form collective modes through mode coupling. Theorem 5.1 gives the collective-mode phase boundary: "The origin is locally unstable along a collective mode iff λmax(C⊤(µI + λ2 ΓE)(−1)C) > µ. Corollary 5.2 establishes that matrix R4, unlike R2, directly penalizes predictor-space leakage: w1 = w2 need not imply β1 = β2 because βi = wi θi. The paper notes that oracle ΓE is not uniquely identifiable from observational associations in finitely many training environments alone."

Regularization-Path Predictions Summary:

Table 1 summarizes the predicted regularization phenotypes. R2 and R2R4 methods (including MMD-mean, CORAL-full, IGA) Move the local onset and enable finite-strength mode elimination. R4 methods (CORAL-cov, Fishr, V-REx) Continuously shrink an acquired mode. IRM and Gaussian MMD May undergo global collapse at large regularization strength. GroupDRO Suppresses the shortcut from small regularization strengths, but may retain a nonzero shortcut plateau. DANN Produces non-monotonic paths, oscillations, or large seed variability. Fish/MLDG Shows direction-dependent R2-like suppression at small strength and becomes unstable at large strength. ERM and Mixup have No sign-controlled shortcut suppression.

Experiments Summary:

The experiments test the predictions in sequence. Figure 2 shows that R2 determines whether a mode can grow from the origin, while R4 mainly controls the amplitude after growth. Figure 3 shows that objective-specific low-order coefficients derived from each learning objective are inserted into the same reduced Landau equilibrium, yielding accurate loading predictions for multiple methods. Figure 4 calibrates critical strengths, confirming the distinct effects of R2 and R4: adding R4 changes the amplitude and near-critical slope of the loading branch without shifting the R2-controlled critical point. Figure 5 shows full regularization paths, where OOD AUC improves only when shortcut suppression preserves the stable mode. Figure 6 shows that the qualitative regularization phenotypes of the bilinear model persist, while transition scales and large-strength tails shift with depth in ReLU networks. Figure 7 demonstrates that coupled matrix R2, R4, and their combination agree with the spectral theory: they suppress the shortcut while preserving the collective mode and improving OOD AUC.

Conclusion Summary:

The paper concludes: "We develop a Landau-theoretic framework for predicting how invariant-learning objectives control the onset, suppression, and retention of predictive modes from their low-order structure. The resulting objective signatures organize existing methods by distinct regularization phenotypes and extend naturally from individual features to collective modes. Future directions include improving data-driven estimation of the environmental sensitivity operator ΓE and characterizing its identifiability from finite environments, extending the framework to deeper and more structured nonlinear architectures with richer feature interactions, and developing a fuller theory of learning dynamics beyond the local near-critical regime studied here."

Improvements to AI Systems:

1. Predictive Regularization-Path Planning: AI systems can now predict, before training, whether a given invariant-learning objective (e.g., IRM, V-REx, CORAL, Fishr) will eliminate a spurious feature at finite regularization strength, continuously shrink it, or cause collapse. This enables automatic selection of the optimal regularization method and strength for a given dataset, avoiding trial-and-error.

2. Selective-Retention Window Tuning: Using the derived critical strengths (λ2,c for shortcut vs. stable modes), AI systems can automatically identify and operate within the selective-retention window—a range of regularization strengths that suppresses shortcuts while preserving stable predictive modes. This directly improves OOD generalization without sacrificing in-distribution accuracy.

3. Objective-Signature-Based Method Selection: AI systems can compute low-order Landau coefficients (r, u) from any candidate invariant-learning objective and use them as signatures to classify methods into phenotypes (R2-like, R4-like, collapse-prone, non-monotone). This allows an AI to choose or even design a new objective with desired regularization behavior (e.g., finite-strength elimination vs. continuous shrinkage) for a specific task.

4. Early-Stopping and Instability Detection: The theory predicts non-monotone tails, oscillations, and collapse at large regularization for methods like DANN, Fish/MLDG, and IRM. AI systems can monitor the effective quadratic coefficient r(λ) during training and automatically halt or adjust regularization before entering unstable regimes, preventing degenerate solutions.

5. Mode-Coupling-Aware Regularization: For multi-feature or multi-modal data, the collective-mode phase boundary (Theorem 5.1) enables AI systems to design regularization that targets shortcut modes in predictor space (via matrix R4) rather than representation space, directly penalizing leakage between components (e.g., w1=w2 but β1≠β2). This improves fairness and robustness in multi-task or multi-environment settings.

6. Data-Efficient Environment Sensitivity Estimation: The paper highlights that ΓE is not uniquely identifiable from finite environments. AI systems can now use the Landau framework to design active learning or environment-augmentation strategies that estimate ΓE more reliably, improving the accuracy of all downstream regularization predictions.

7. Architecture-Aware Regularization Transfer: The validation on ReLU networks shows that qualitative phenotypes persist but transition scales shift with depth. AI systems can use this to automatically rescale regularization strengths when changing network depth or width, maintaining desired suppression behavior without re-tuning.

8. Closed-Form Steady-State Loading Prediction: Given a dataset and objective, AI systems can compute closed-form steady-state loadings (β*) for each mode without full training. This enables rapid model selection, hyperparameter sweeps, and theoretical guarantees on final feature magnitudes—useful for interpretability and debugging.

9. Collapse-Avoidance for Large-Regularization Regimes: For methods prone to global collapse (IRM, Gaussian MMD), AI systems can automatically cap regularization strength based on the predicted collapse boundary, ensuring stable training even under aggressive regularization schedules.

10. Coupled-Mode Regularization for Matrix Extensions: For high-dimensional or structured outputs (e.g., multi-label, multi-task), AI systems can apply the spectral theory to jointly regularize coupled modes, preserving collective predictive patterns while suppressing environment-specific ones—improving OOD performance in complex, correlated feature spaces.