Correlational Training of Morphological Neural Networks

arXiv:2610.11740 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Correlational Training of Morphological Neural Networks".

Jane: The gist Neural networks are typically trained using first-order methods and back-propagation,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We were talking about the paper "Correlational Training of Morphological Neural Networks." We covered how they propose a weight update rule inspired by Multiplicative Weights Update to help train these complex networks.

Jane: They claim this method uses a correlation-based reward that favors inputs aligned with the desired output change, even when strong gradient signals aren't present.

Lu: The core idea is viewing each morphological perceptron as an instance of the learning from experts’ advice problem in log-space and using a Taylor expansion to create a linear surrogate for the signal of desired change in pi.

Meng: They are essentially measuring whether different expert opinions vary across the batch in the same direction as what you want that layer output to change.

Tom: That leads to an update rule that is mathematically equivalent to an exponentiated-gradient and Multiplicative Weights Update scheme, which they call correlational training.

Jane: And they claim this approach works by using a reward model constructed from that Taylor expansion, which measures the correlation between the expert advice and the target change.

Lu: They evaluate this method empirically by training fully connected layers both as standalone models on simple classification tasks and as components of larger transformers.

Meng: The main implication for us is that when we encounter morphological layers whose weight Jacobians are sparse, standard back-propagation might be suboptimal for these architectures. This paper gives us a concrete alternative update rule to consider for those specific architectures.

Tom: It’s about finding better ways to train these specialized networks where the natural gradient signals are poor, offering a way to boost performance and stability compared to ordinary back-propagation on certain tasks.

Jane: Ultimately, the work shows that we can use correlation as a signal for weight updates, which helps morphological neural networks perform better and more reliably than they might otherwise.

Lu: But they also point out that the method is equivalent to using the reward r prime i equals Cov(˜dy(x;w,c), xi) because the expected change term is zero among all experts, which justifies it being called correlational training.

Meng: That makes sense from a theoretical standpoint, showing why this update rule behaves in a certain predictable way regardless of what the specific expert advice is.

Tom: And on the practical side, they showed this approach yields improvements on eight out of nine benchmarks across various tasks, with gains up to thirty-two point eight four percentage points on some of them.

Jane: That's a solid result, especially when you consider that standard training methods often struggle with the sparse gradient signals in these morphological structures.

Lu: They also mention that for hard morphology, where tau equals zero, they use the zero-temperature limit where those surrogate rewards update the morphological weights additively before applying max normalization.

Meng: So for an engineer building these layers, this suggests we might not need perfect gradient signals to get useful training progress on these kinds of models.

Tom: Exactly, and they showed that it remains stable across runs on some datasets like MNIST and CIFAR-ten even showing a reduction in variance.

Jane: So what does this mean for the people using these networks? It means they can expect better performance and much less headache when dealing with those tricky layers.

Lu: This opens up a path to designing architectures where the training process itself is more guided by consensus rather than just raw mathematical derivatives.

Conclusion: Tom: We're wrapping up our talk on "Correlational Training of Morphological Neural Networks." This paper focuses on how they improve the training dynamics for these specific networks compared to standard back-propagation.

Jane: Right, it’s essentially a new way to update the network weights by looking at how the desired output change correlates with what the experts are suggesting.

Lu: The core idea is using something inspired by Multiplicative Weights Update, which means instead of just following the steepest gradient, we favor updates that move us in a direction that experts agree is good.

Meng: So, for someone building these morphological layers, this suggests we might not need perfect gradient signals to get useful training progress.

Lalam: If you think about it through the lens of culture and learning, it means the network learns more robustly because it's not getting stuck on just one noisy signal; it’s looking at the overall trend from different expert viewpoints.

Tom: Exactly, and they showed this actually works for image classification on eight out of nine benchmarks, with gains up to thirty-two point eight four percentage points on some of them.

Jane: That's a solid result, especially when you consider that the standard training methods often struggle with the sparse gradient signals in these morphological structures.

Lu: The paper points out that while this correlation method is powerful, there's still a limit—the hard morphology case where everything becomes extremely sparse and optimization gets really tough.

Meng: From an engineering standpoint, it's interesting because it offers a more stable training dynamic, reducing that run-to-run variability we often see in these kinds of models.

Tom: Yeah, they did mention that while this correlation method is powerful, there's still a limit—the hard morphology case where everything becomes extremely sparse and optimization gets really tough.

Jane: So what does this mean for the people using these networks? It means they can expect better performance and much less headache when dealing with those tricky layers.

Konstantinos Fotopoulos, Petros Maragos

HERON - Hellenic Robotics Center of Excellence · Institute of Robotics, Athena Research Center, Marousi, Greece · School of ECE, National Technical University of Athens

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Preprint

Code: https://github.com/kostfoto/Morph-MWU

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

The gist: The gist Neural networks are typically trained using first-order methods and back-propagation, but this approach may be suboptimal for morphological layers whose weight Jacobians are sparse and whose

Key concepts

Morphological Networks: These are neural networks based on mathematical morphology operations like dilation and erosion instead of standard matrix multiplications. They replace addition with maximum operations and multiplication with addition, making them potentially more efficient for inference.
First-Order Optimization Limitations: Methods like back-propagation rely on first-order gradients, which are often sparse or uninformative for morphological layers. This makes training difficult because the gradient signals do not provide enough useful direction to update the weights effectively.
Correlation-Based Reward: The proposed method uses a reward system based on expert advice in logarithmic space. It measures how much different experts' advice varies across a batch in the same direction as the desired change of the layer output, guiding weight updates toward better solutions.

Summary

The gist Neural networks are typically trained using first-order methods and back-propagation, but this approach may be suboptimal for morphological layers whose weight Jacobians are sparse and whose resulting parameter gradients can be poor<ref:2610.11740#pg2>. This work proposes a novel weight update method inspired by the Multiplicative Weights Update (MWU) scheme to address these challenges in training morphological neural networks<ref:2610.11740#pg4>.

Introduction and Motivation

First-order optimization methods such as stochastic gradient descent and back-propagation have enabled efficient training of typical neural networks predominantly based on matrix multiplications combined with nonlinear activations<ref:2610.11740#pg2>. It is unclear whether this approach is optimal for alternative architectures with sparse or otherwise poorly informative parameter Jacobians<ref:2610.11740#pg4>. Morphological networks are of particular interest due to their lack of multiplications—making them potentially more inference-efficient than their linear counterparts—and their rich supporting theoretical foundation<ref:2610.11740#pg4>. These networks are based on the morphological operations of dilation and erosion, replacing addition by maximum and multiplication by addition<ref:2610.11740#pg3>. Morphological networks are notoriously difficult to train because of the sparsity of their gradient signals<ref:2610.11740#pg4>. This difficulty stems from the sparsity of their gradient signals<ref:2610.11740#pg4>, and the non-informativeness of the gradient is also a limiting factor<ref:2610.11740#pg4>.

Proposed Method

The authors propose a correlation-based reward that favors inputs aligned with the desired output change, regardless of whether a strong gradient signal has reached their weight<ref:2610.11740#pg4>. The method frames the training of weights as a learning from experts’ advice problem in logarithmic space<ref:2610.11740#pg4>. Specifically, the reward model is constructed using a Taylor expansion to yield a linear surrogate of the preconditioned signal of desired change in pi<ref:2610.11740#pg4>. The reward measures whether expert i’s advice varies across the batch in the same direction as the desired change of the layer-output<ref:2610.11740#pg4>. This leads to an update rule for log-weights that is equivalent to an exponentiated-gradient/MWU scheme<ref:2610.11740#pg4>. For hard morphology (τ = 0), the method uses the corresponding zero-temperature limit, where the surrogate rewards update the morphological weights additively, followed by maxnormalization wi ← wi − maxj wj<ref:2610.11740#pg4>.

Experimental Evaluation

The approach was empirically evaluated by training fully connected layers both as standalone models and as parts of larger transformer networks<ref:2610.11740#pg4>. Across nine benchmarks, correlational training yields improvements on eight, by up to 32.84 percentage points, while substantially reducing run-to-run variability<ref:2610.11740#pg4>. Table 1 reports test performance across all benchmarks, showing that correlational training improves the morphological model over vanilla back-propagation on eight of the nine tasks, with gains ranging from 5.27 to 32.84 percentage points<ref:2610.11740#pg5>. The largest gains occur on MNIST (+32.84 pp), Fashion-MNIST (+23.37 pp), Mushroom (+23.31 pp), Chess (+19.19 pp), and Adult (+18.99 pp)<ref:2610.11740#pg5>.

Training Dynamics and Limitations

Figure 1 illustrates the optimization behavior on MNIST and CIFAR-10, showing that correlational training rapidly reaches high accuracy and remains stable across runs<ref:2610.11740#pg5>. The reduction in variance is also apparent across several PMLB datasets; for example, the standard deviation decreases from 12.52 to 1.08 percentage points on Adult and from 21.35 to 2.57 on Mushroom<ref:2610.11740#pg5>. A remaining challenge is the hard limit τ → 0, where morphology becomes increasingly sparse and harder to optimize<ref:2610.11740#pg5>. The authors note that while the method is beneficial in this regime, on image classification the softened τ = 1 setting was more competitive<ref:2610.11740#pg5>.

Conclusion

The proposed correlation-based weight update rule for morphological neural networks substantially improves both performance and training stability over ordinary back-propagation, with gains on eight of the nine tested benchmarks. While conventional MLP and ViT baselines remain stronger on most tasks, our results indicate that poor first-order trainability can substantially underestimate the practical capability of morphological architectures.

--- Page 2 ---

The vector set R n, equipped with the partial order x⪯ y ⇔ xi ≤ yi, ∀i, forms a complete lattice<ref:2610.11740#pg3>. (Anti-)dilations and (anti-)erosions from R n to R can be defined as δw(x) = n max i=1 (xi + wi), εm(x) = n min i=1 (xi + mi), δa m(x) = n min i=1 (−xi + mi), εa w(x) = n max i=1 (−xi + wi)<ref:2610.11740#pg3>. Combining these building blocks as perceptrons, we can construct more complex functions, guaranteed by the representation theory of mathematical morphology [8, 12]<ref:2610.11740#pg3>. if f: R n → R is a 1-Lipschitz function, we can write it as a supremum over a combined erosion and anti-dilation: f(x) = sup r∈Rn f(r) + (εr(x) ∧ δa−r(x))<ref:2610.11740#pg3>. By discretizing the representation theorem, we obtain an approximation of f in the form of a Min-max-plus network [10]: f(x)≈max i f(ri)+(min j (xj + ri,j) ∧ min j (−xj − ri,j))<ref:2610.11740#pg3>. Training such a network using gradient descent can be troublesome due to the sparsity of the gradient signal during backpropagation and the non-first-order universality of the network [14, 15, 11]<ref:2610.11740#pg3>.

--- Page 3 ---

Consider a dilation δw˜ (x) = Wn i=1(xi + ˜wi) with inputs x ∈ R n and weights w˜ ∈ R n, as in (1). Its subdifferential with respect to the weights is given by ∂δw˜ (x) = conv(e i: i ∈ argmaxi (xi + ˜wi))<ref:2610.11740#pg4>. δw˜ (x) is a Lipschitz continuous function, and thus differentiable almost everywhere, and precisely when the argmax in the equation of the subdifferential is a singleton. In this case, all but one weights receive a gradient of 0, and optimization using gradient descent becomes inefficient<ref:2610.11740#pg4>. Notice that a dilation as in (1) is essentially a competition among the inputs, with some a priori bias given to each input in the form of a weight wi<ref:2610.11740#pg4>. Our method is motivated by the Maslov dequantization of this maximum operator<ref:2610.11740#pg4>. We consider functions of the form yτ (x, w, c) = c + τ logXn i=1 pie e xi/τ, pi = e wi/τ, with weights w such that P i pi = P i e wi/τ = 1<ref:2610.11740#pg4>. Here, the set of functions f i n i=1 xi function as experts’ opinions in log-space<ref:2610.11740#pg4>. The distribution p i n i=1 is a distribution over the inputs, with w i n i=1 the experts’ weights in log-space<ref:2610.11740#pg4>.

Improvements for AI systems

No first-order optimization methods such as stochastic gradient descent and back-propagation are used for training morphological layers; instead, a novel weight update method for morphological neural networks inspired from the Multiplicative Weights Update (MWU) scheme is proposed that rewards inputs that correlate well with the desired change in the output, regardless of whether they see a strong gradient signal.
The improved AI system can effectively train complex morphological architectures like Min-max-plus networks and MPM networks by bypassing issues arising from the sparsity of their gradient signals, leading to substantial performance gains, such as up to 32.84 percentage points on benchmarks like MNIST.
Training dynamics are stabilized across different runs because the method substantially reduces run-to-run variability; for instance, the standard deviation in accuracy on Adult data decreased from 12.52 to 1.08 percentage points.
The system exhibits improved convergence speed and stability compared to vanilla Adam, as shown by the improvement of the MorphViT performance on CIFAR-10 from 63.96 ± 0.88% under vanilla training to 70.20 ± 0.62% under the proposed correlational update.
The system can handle hard morphology (where τ = 0) by using a corresponding zero-temperature limit that allows morphological weights to be updated additively, followed by maxnormalization wi ← wi − maxj wj.