Convergent Evolution in Algorithmic Space

The paper Convergent Evolution in Algorithmic Space by Patrick Krauss et al. investigates whether independently initialized neural networks trained on the same task converge toward similar internal weight structures, analogous to convergent evolution in biology. The authors introduce a matching-based framework to compare multilayer perceptrons (MLPs) in structural weight space, addressing the neuron permutation problem where hidden neurons can be arbitrarily permuted without changing the network's function.

The key methodological contribution is a two-stage alignment procedure: first, hidden neurons are coarsely aligned using permutation-invariant features (PIFs) such as signed total input, number of nonzero connections, number of positive connections, and Euclidean norms of incoming and outgoing weight vectors. Second, this alignment is refined by iterative Hungarian matching, where cost matrices are constructed from distances between incoming–outgoing neuron profiles. The authors define three structural distance metrics: Euclidean distance, Structural Network Dissimilarity (SND), and Alpha distance. The Alpha distance, which emphasizes sign mismatches of jointly large weights, proved most effective at revealing task-specific convergence.

The experiments use small MLPs with architecture 784 → 64 → 10 trained on MNIST, Fashion-MNIST, and KMNIST, with three independently initialized networks per task (nine total). The results show that networks trained on the same task remain closer to one another than to networks trained on different tasks. Specifically, with the Alpha metric, all six distances decrease rapidly during the first training epoch and then continue to decline much more gradually, and a clear separation between the within-task and between-task curves emerges. Multidimensional scaling visualizations confirm that three distinct clusters have already emerged after one training epoch, corresponding to the three tasks, and these clusters become progressively more compact during training.

A particularly striking finding concerns the earliest phase of training. The authors observe that Classification accuracy rises rapidly before the matched structural distances show strong task-specific separation, and before the global weight distribution visibly changes. At approximately E = 0.06, MNIST accuracy has already reached 0.623, yet the structural distances remain nearly constant. The paper explains: the weights undergo subtle but functionally important changes that strongly improve performance while remaining largely undetected by the Alpha distance metric. This suggests early learning may first operate through subtle, distributed adjustments that strongly affect function while leaving coarse network morphology almost unchanged.

Ablation experiments support the importance of large-magnitude weights: nearly 90% of the weakest weights can be removed with little effect on test accuracy in the first layer, while removing only 10% of the strongest weights already causes a dramatic collapse in performance. Tracking individual weights reveals that most of the weights that ultimately become task-relevant already possess their final sign in the randomly initialized network at epoch 0, and these are progressively amplified during training, whereas weaker weights frequently change sign before eventually being suppressed toward small magnitudes.

The paper concludes with a two-stage view of training: "In the earliest phase, the network undergoes a highly effective functional calibration: tiny, directed changes in weights move many samples toward or across correct decision boundaries. Only later does a clearer structural reorganization become visible. The authors interpret this as evidence that the initial random network is not an inert blank slate and that the landscape may therefore be effectively 'target-rich': a random initialization need not be close to one unique canonical solution, but may already lie within the basin of attraction of one of many equivalent or nearly equivalent realizations."

The discussion positions this work relative to representation similarity studies (SVCCA, CKA), permutation symmetry research (Git Re-Basin), and the Platonic Representation Hypothesis, emphasizing that the paper shifts the question of convergence from activation space to parameter space. Future directions include scaling to larger networks, constructing datasets with explicit hierarchical structure, dedicated studies of the early learning phase, mapping attractor basins, and exploring language-processing attractors.

Improvements to AI Systems:

1. Early-Training Diagnostic Module: Implement a lightweight monitoring layer that tracks Alpha-distance metrics (sign-mismatch-weighted structural distance) between independently initialized copies of the same network during the first few hundred gradient steps. This allows AI systems to detect whether they are in the functional calibration phase (subtle weight changes, high accuracy gain, low structural divergence) versus the structural reorganization phase. The system can then adaptively adjust learning rates or switch optimization strategies—e.g., using larger, more aggressive updates once structural separation begins, or employing weight-magnitude-aware pruning earlier to preserve the critical large-magnitude weights identified as task-relevant.

2. Sign-Preserving Initialization and Amplification Scheduler: Since the paper shows that most task-relevant weights already have their final sign at initialization and are progressively amplified, AI systems can be improved by (a) initializing weights with a wider distribution of signs to increase the chance of pre-existing correct signs, and (b) implementing a sign-locking mechanism that penalizes sign flips for weights above a certain magnitude threshold, while allowing weaker weights to flip freely. This reduces wasted computation on sign changes and accelerates convergence toward task-specific attractors.

3. Structural Convergence-Based Ensemble Selection: Use the Alpha distance to compare multiple independently trained models on the same task. Instead of ensembling all models, the system can select a diverse subset that maximizes within-task structural similarity (indicating they have converged to the same attractor basin) while minimizing redundancy. This yields smaller, more efficient ensembles with higher predictive confidence, as the selected models are known to share robust task-relevant weight structures.

4. Task-Similarity-Aware Transfer Learning: The finding that networks trained on different tasks form distinct clusters in weight space after just one epoch suggests that structural distance can serve as a fast, parameter-space-based measure of task relatedness. An AI system can use this to pre-select source models for transfer learning: given a new task, it computes Alpha distances between the new task's early-training network and a library of pre-trained networks, then transfers weights from the closest source—reducing fine-tuning time and improving generalization by starting from a structurally aligned initialization.

5. Target-Rich Landscape Exploitation via Multi-Start Search: The paper's conclusion that random initializations lie within basins of attraction of many equivalent solutions implies that AI systems should not waste resources on exhaustive hyperparameter tuning for a single solution. Instead, the system can launch multiple short, parallel training runs (e.g., 1–2 epochs), compute their Alpha-distance-based structural clusters, and then continue training only the most representative run from each distinct cluster. This basin sampling strategy explores multiple near-optimal solutions simultaneously, improving robustness and avoiding premature convergence to a single local minimum.

6. Adaptive Pruning Based on Sign Stability and Magnitude: The ablation results show that removing 90% of weakest weights is harmless, but removing 10% of strongest is catastrophic. An improved AI system can implement a pruning schedule that: (a) tracks sign stability of each weight across early epochs, (b) permanently freezes and amplifies weights that are both large-magnitude and sign-stable, (c) progressively suppresses weights that frequently flip sign (as they are likely task-irrelevant), and (d) prunes the bottom 90% of remaining weights only after structural convergence is detected via Alpha distance. This yields sparser networks with preserved task-critical structures.

7. Early-Stopping Criterion Based on Structural Saturation: Instead of relying solely on validation accuracy, the AI system can monitor the rate of change of within-task Alpha distances. When the distance between two independently initialized copies of the network stops decreasing (i.e., structural saturation), training can be halted—even if accuracy is still marginally improving—because further updates are unlikely to yield new task-relevant structure. This saves compute and prevents overfitting to noise in the later, slower phase of training.

8. Cross-Task Regularization via Structural Anchoring: During training on a new task, the system can compute Alpha distances to a set of reference networks from known tasks. By adding a regularization term that penalizes structural drift toward unrelated tasks (e.g., increasing distance to MNIST-trained networks when training on KMNIST), the system is guided to stay within its task-specific attractor basin, improving generalization and reducing catastrophic forgetting in continual learning settings.

What the Improved AI System Can Do:

• Self-diagnose its learning phase (calibration vs. reorganization) and adapt its optimization strategy accordingly, leading to faster convergence and better final accuracy.

• Train more efficiently by exploiting pre-existing correct weight signs, avoiding wasteful sign flips, and pruning aggressively without performance loss.

• Build smaller, more reliable ensembles by selecting models that have converged to the same structural solution, reducing inference cost while maintaining or improving accuracy.

• Transfer knowledge more intelligently by using parameter-space similarity (not just activation similarity) to pick source models, enabling faster adaptation to new tasks with less data.

• Explore multiple near-optimal solutions in parallel, reducing the risk of getting stuck in poor local minima and improving robustness to random seeds.

• Stop training at the right time—when structural convergence is achieved—saving computational resources without sacrificing task performance.

• Maintain task-specific knowledge in continual learning by structurally anchoring to the current task's attractor basin, mitigating interference from previously learned tasks.

In evolutionary biology, unrelated organisms can independently evolve similar structures when exposed to similar functional demands. Here we ask whether an analogous form of convergent evolution occurs during neural network training: do networks with different random initializations develop similar internal weight structures when trained on the same task? This question is technically nontrivial because hidden neurons can be arbitrarily permuted without changing the represented function, making direct matrix comparisons misleading. We introduce a matching-based framework for comparing multilayer perceptrons in structural weight space. Hidden neurons are first coarsely aligned using permutation-invariant features and then refined by iterative Hungarian matching. After alignment, networks are compared with structural distance metrics designed to emphasize task-relevant weight patterns. Applying this approach to ensembles of small MLPs trained on MNIST, Fashion-MNIST, and KMNIST, we find that networks trained on the same task remain closer to one another than to networks trained on different tasks. Thus, task-specific training appears to guide initially random networks toward distinct regions, or attractors, in structural network space. The earliest phase of training reveals an additional and unexpected phenomenon. Classification accuracy rises rapidly before the matched structural distances show strong task-specific separation, and before the global weight distribution visibly changes. Nevertheless, individual weight entries already begin to drift in a coordinated manner. This suggests that early learning may first operate through subtle, distributed adjustments that strongly affect function while leaving coarse network morphology almost unchanged. We treat this early morphogenesis as a first glimpse of a richer dynamical process that will be investigated in future work.