Coefficient Calibration as Selection Pressure in Symbolic Regression

The gist In memetic symbolic regression, candidate structures are compared after coefficient calibration, so the calibration protocol itself contributes to evolutionary selection This paper proposes Dirichlet-Sinkhorn Constant Averaging (DSCA), a calibration strategy that partitions the optimization data into equally sized subsets with different covariate distributions, calibrates each candidate independently on every partition, and evaluates it at the mean of the resulting parameters

A New Calibration Strategy

The authors introduce DSCA as a new calibration strategy for symbolic regression that replaces centralized coefficient calibration with calibration over controlled covariate-heterogeneous partitions followed by parameter averaging This modification modifies only the individual-evaluation stage, allowing it to be readily integrated into standard SR pipelines without changing the underlying search procedure or outer model-selection criterion The strategy involves two main stages: once, before the search, the dataset D is used to create K equally sized subsets D1,..., DK with controlled covariate heterogeneity During the search, every candidate is calibrated independently on each subset and the resulting parameter vectors are averaged to obtain the parameterization used for fitness evaluation.

Theoretical Characterization of Selection Effect

The authors characterize the excess loss induced by DSCA relative to centralized calibration and show how this additional calibration-stage penalty can alter evolutionary selection At population level, the excess vanishes for correctly specified and identifiable expressions, whereas under misspecification it can persist when partition-specific calibrations do not aggregate to the pooled optimum The selection-relevant excess loss introduced by DSCA is therefore defined as ∆N′:= LN′ (¯θ) − LN′ (ˆθN′), which measures how much additional empirical loss is assigned to a fixed candidate structure because DSCA uses the average of the locally calibrated parameters rather than the centralized optimum.

Empirical Evaluation and Mechanism

Across controlled synthetic benchmarks and ten real-world datasets, DSCA improves functional recovery and the accuracy–complexity tradeoff over centralized Broyden–Fletcher–Goldfarb–Shanno and Levenberg-Marquardt calibration under selection by negative log-likelihood and by the Akaike and Bayesian information criteria Mechanism analyses associate the DSCA excess loss with the generalization gap and show that the effect is not reproduced by repeated centralized fitting. The results indicate that controlled heterogeneous calibration provides a complementary source of selection pressure in symbolic-regression search.

Key Findings on Performance and Complexity

DSCA achieves higher functional-recovery rates than centralized BFGS and Levenberg-Marquardt on the synthetic benchmarks and a more favorable accuracy–complexity tradeoff on the real-world datasets Its advantage is clearest in low-sample regimes, where sample-specific calibration and overfitting are most likely, and narrows as the sample size grows. After a common centralized re-fit, structures retained by DSCA exhibit smaller generalization gaps, and ∆N′ is positively associated with this gap across structures and algorithms of origin. Furthermore, the configuration used in the main experiments, α = 1 with K = 5, gives up a small amount of R2 relative to a pure restart strategy in exchange for a reduction of roughly 2 to 15 nodes.

Future Directions

Two directions appear particularly important for future work: First, symbolic simplification removes exact algebraic redundancies but does not prevent weakly identified or nearsingular parameterizations, and structural repair operators that eliminate such degeneracies before calibration could make parameter averaging more reliable Second, the interaction between heterogeneous calibration and outer selection criteria such as AIC and BIC is explored only empirically here, characterizing how they jointly shape evolutionary pressure remains an important direction for further study.

The paper introduces Dirichlet-Sinkhorn Constant Averaging (DSCA), a coefficient-calibration strategy for genetic programmingbased SR that repurposes distributed parameter estimation as a source of selection pressure during evolutionary search This modification modifies only the individual-evaluation stage, allowing it to be readily integrated into standard SR pipelines without changing the underlying search procedure or outer model-selection criterion.

How it works

The strategy involves replacing a single calibration step with two stages: once, before the search, the dataset D is used to create K equally sized subsets D1,..., DK with controlled covariate heterogeneity During the search, every candidate is calibrated independently on each subset and the resulting parameter vectors are averaged to obtain the parameterization used for fitness evaluation.

Surrogate-Based Dirichlet-Sinkhorn Partitioning

The partitioning procedure is guided by four main parameters: i) the fraction ρ of D which is used to define the covariate function driving sample stratification for allocation, ii) the number of subsets K, iii) the number of strata S, and iv) the concentration parameter, α, which controls data heterogeneity across partitions This procedure involves Surrogate Scoring (steps 1-3), Stratification (step 4), Dirichlet Sampling (step 5), Matrix Balancing (steps 6-10) using the Sinkhorn-Knopp algorithm, and Integer Rounding/Slicing to yield K partitions.

Local Optimization and Averaging

Before numerical calibration, each candidate expression f(·, θ) is simplified symbolically to remove exact algebraic redundancies and structurally non-identifiable parameterizations During the evolutionary search, f(·, θ) is calibrated independently on each of the K generated partitions using a gradient-based optimizer like BFGS to yield K locally calibrated parameter vectors ˆθ1,..., ˆθK These sets of local constants are aggregated by computing their arithmetic mean, which produces a single, averaged set of constants for the candidate expression: ¯θ = 1/K Σ K k=1 ˆθk.

Theoretical Analysis of DSCA Implications

The analysis isolates the excess loss that is directly relevant to evolutionary selection, defined as ∆N′:= LN′ (¯θ) − LN′ (ˆθN′) This quantity measures how much additional empirical loss is assigned to a fixed candidate structure because DSCA uses the average of the locally calibrated parameters rather than the centralized optimum. The analysis shows that ∆N′ vanishes for correctly specified and identifiable expressions, whereas under misspecification it can persist when environment-specific calibrations do not aggregate to the pooled optimum

Finite-Sample Connection

Under standard regularity conditions and assuming identifiable models, fixed K, and large N′, the empirical excess fluctuates around a non-vanishing population quantity ∆N′ Specifically, under correct specification the entire DSCA excess is a second-order finite-sample effect, whereas when ∆ > 0 a nonzero population excess persists and only its sampling fluctuations vanish at a slower rate.

Surrogate Informativeness

A surrogate is informative when its ordering separates regions of covariate space in which the preferred calibration differs This occurs when the environment weight Kπk(g(X)) is not aligned with regions in which the candidate has a nonzero conditional loss gradient.

1. textbfThe Dirichlet-Sinkhorn Constant Averaging (DSCA) calibration strategy enhances evolutionary selection pressure by modifying how candidate structures are evaluated, specifically by introducing a penalty for solutions whose empirical performance relies on highly sample-specific coefficient calibration. This mechanism is expected to reduce the generalization gap for misspecified models.

2. textbfImproved generalization under misspecification by exploiting covariate heterogeneity through DSCA's mechanism where misspecified models can incur a persistent penalty that, in the near-homogeneous perturbative regime, scales with partition heterogeneity. This suggests that controlled heterogeneity provides a complementary source of selection pressure in symbolic-regression search against overfitting.

3. textbfEnhanced accuracy-complexity tradeoff by favoring simpler expressions through DSCA's selection effect on structure size, as shown by the result: DSCA tends to favor smaller expressions while retaining competitive predictive accuracy. This implies that DSCA offers a more favorable model selection mechanism than standard methods like centralized Broyden–Fletcher–Goldfarb–Shanno.

4. textbfRobust performance in low-sample regimes through sample-specific calibration, as DSCA improves functional recovery and the accuracy–complexity tradeoff over centralized Broyden–Fletcher–Goldfarb-Shanno and Levenberg-Marquardt calibration when the advantage is clearest in low-sample regimes. This makes the system more reliable when data is scarce.

5. textbfAdaptive model selection guided by selection criteria, as DSCA's effect is not reproduced by repeated centralized fitting, indicating that the structure favored by DSCA is distinct from those retained by standard methods, allowing for a mechanism where controlled heterogeneous calibration provides a complementary source of selection pressure in symbolic-regression search. This suggests better guidance when using criteria like AIC or BIC.