Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers

arXiv:2606.02927 · cs.CV · Submitted 2026-06-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Geometry, Not Statistics".

Jane: Here's the gist of the paper. Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks <ref:2606.02927#pg2> This work demonstrates that they can be fully replaced by a single learnable activation mechanism <ref:2606.02927#pg2> The paper introduces SALU, a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters <ref:2606.02927#pg2> Building on SALU, the authors propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity <ref:2606.02927#pg2>.

Tom: The Core Concept of Total Plasticity The central idea is that all major components of signal propagation become fully learnable through geometric plasticity <ref:2606.02927#pg2> This means connections are learned via network weights, stabilization is learned through SALU-based adaptive saturation, and gating dynamics are learned via SWALU and GALU <ref:2606.02927#pg2>.

Jane: This framework aims to replace external statistical constraints with intrinsic neuron-level dynamics <ref:2606.02927#pg2>.

Tom: The SALU Activation Mechanism SALU is defined as SALU(x; a, b) = ax√1 + abx2 where a > 0, b > 0 <ref:2606.02927#pg2> It acts as a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics <ref:2606.02927#pg2>.

Title and authors: Tom: So we're looking at this paper titled "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," and what it’s doing is replacing those big normalization layers like Batch Normalization with a single learnable activation mechanism.

Jane: That sounds complicated. Can you break down what the authors are actually proposing without all the math?

Lu: Basically, they say that instead of using things like Batch Normalization to stabilize training by looking at batch statistics, we can use something called SALU which is a bounded activation function that handles stabilization internally through its own shape and scale.

Meng: So it’s trying to get rid of the need for those external statistical constraints, which means no more needing batch statistics or extra affine parameters during training? Because from my point of view, relying on batch stats is just adding more dependencies that break when you move to a different setup.

Lalam: The core idea they are pushing is called total plasticity, where everything—the connections and the stabilization and the gating dynamics—becomes fully learnable through this geometric approach.

Tom: That’s a big claim. So what exactly is this SALU function they introduce? What does it look like mathematically?

Jane: The paper defines it as SALU of x with a and b, which is written as a times x root one plus a times b times x squared, where 'a' and 'b' are positive numbers.

Lu: That specific formula, SALU(x; a, b) = ax root one plus abxtwo with 'a' greater than zero and 'b' greater than zero, is what provides that bounded activation function they are using to stabilize the signal geometry <ref:2606.02927#pg1,SALU(x; a, b) = ax>.

Meng: And they claim this approach avoids relying on batch statistics or external affine parameters, so it’s not adding any new statistical overhead during the forward pass? That would be a huge win for deployment because we don't have to track those running means or variances.

Lalam: The paper shows that when you use standard normalization layers, like Batch Normalization, the learnable parameters in activations like PReLU rapidly lose their meaningful adaptability after about ten epochs on CIFAR-ten data because the normalization dominates the statistical dynamics there <ref:2606.02927#pg2>.

Tom: Wait, so they show that this plasticity suppression effect happens when you pair those learnable activation parameters with normalization layers, which is what makes them lose adaptability quickly?

Jane: Exactly. The authors point out that without Batch Normalization, the learnable slope in PReLU keeps drifting significantly throughout training because it’s highly adaptive but unstable.

Lu: In contrast, when you use SALU for stabilization instead of Batch Normalization, the geometry of the activation stays stable yet adaptive, which is what they call geometric plasticity.

Title and authors: Meng: So they are showing that the learning process can be controlled by how the activation function's shape and saturation change dynamically during training instead of relying on external layers to do that work.

Lalam: They build on SALU by introducing two more gated versions, SWALU and GALU, which are basically SALU-based extensions of Swish and GELU styles, making the gating behavior fully learnable too.

Tom: SWALU is a SALU extension of Swish where they replace the sigmoid-based gating with a SALU-based gate, meaning the gate itself adapts its shape and gain during training.

Jane: And GALU is a generalization of GELU-style gating specifically designed for transformer and attention based architectures, which means it’s tailored for those complex models.

Lu: The authors argue that SALU does more than just stabilize the signal; it acts as a learnable gate whose shape, gain, and saturation all adapt during training through geometric plasticity.

Meng: That sounds like a lot of things being learned at once—the stabilization, the gating dynamics, and the non-linear modulation—all emerging from that single bounded adaptive principle.

Lalam: They frame this as total plasticity where weights are learnable, and stabilization and gating are also learned through this mechanism instead of being imposed externally.

Tom: So what’s the practical takeaway here for someone building a deep network? Are we supposed to ditch all normalization layers entirely just because we can replace them with SALU?

Jane: Well, they show that SaluNet, which uses this framework, achieves state of the art performance on CIFAR-ten and CIFAR-one hundred without any explicit normalization layers.

Lu: They report that SaluNet-C-eighteen gets ninety-seven point three five percent accuracy on CIFAR-ten and eighty-three point two five percent on CIFAR-one hundred and SaluNet also hits competitive accuracy on ImageNet-1K within a standard ninety epoch budget, which is quite impressive.

Meng: That performance is good, but I’m curious about how it handles things at scale. Does this work maintain strong performance even when we test it at the unitary batch limit, meaning batch size one?

Lalam: Yes, the system exhibits extreme robustness to batch size scaling and maintains strong performance even at the unitary limit where normalized architectures usually fail to converge.

Tom: That addresses a major pain point for me because training stability often collapses when you drop the batch size down to one. So this method seems designed specifically to keep things stable under those tough conditions.

Jane: It suggests that instead of relying on batch statistics to smooth out the learning process, we can rely on intrinsic neuron level dynamics to achieve that stability.

Title and authors: Lu: They also found something interesting when they analyzed the learned geometry: there’s a depth dependent stratification where early layers show strong saturation and narrow linear regimes, but deep layers prioritize wide linear regimes and relaxed gating for more representational capacity.

Meng: So it means the network naturally learns how to behave differently in its different parts based on their depth, which is a very interesting way for the AI to structure itself without us having to manually design those layer behaviors.

Lalam: They conclude that this shift from aggregate batch statistics to intrinsic neuron level dynamics offers four fundamental advantages, including geometric plasticity and organic self normalization.

Tom: So we’re talking about Geometric Plasticity and Organic Self-Normalization as the main benefits derived from this SaluNet framework?

Jane: That’s right. It suggests that normalization layers might actually be suppressing the total plasticity that lets deep networks learn effectively in the first place, which is a pretty big observation.

Lu: The authors suggest that shifting this stabilization burden to intrinsic neuron level dynamics gives us geometric plasticity and organic self normalization as key gains for training deep networks without external constraints.

Meng: From a deployment standpoint, if we can achieve that kind of inherent stability without needing those complex statistical layers, it simplifies the entire pipeline significantly.

Lalam: The paper shows that SaluNet is also applicable across different architectures, like Vision Transformers, suggesting this stabilization strategy generalizes beyond just standard convolutional networks.

Tom: So if I’m looking at a CNN or a transformer architecture, I can use this single principle of total plasticity to handle both the stabilization and the gating mechanisms?

Jane: That’s the point; it unifies stabilization and activation into one component, meaning you can use SWALU for Swish-like models or GALU for transformer models under the same framework.

Lu: The paper demonstrates that SaluNet-T, which is applied to Vision Transformers, shows this stabilization strategy works across diverse architectures beyond convolutional networks.

Meng: If it scales well and handles different architectures, that makes it much more attractive for production environments where we have many different model types to maintain.

Lalam: The authors also flag some limitations; they mentioned that the framework can be sensitive to abrupt learning rate schedules, requiring a short adaptation period for the intrinsic parameters 'a' and 'b' to re-adjust.

Tom: And there’s another one too—the paper says that when you are jointly optimizing both the weight parameters and these activation parameters, you need sufficient sample diversity for geometric plasticity to converge effectively.

Jane: So, while the performance numbers are strong, the authors warn us that we still have to be careful about how we schedule learning rates and how diverse our training data is if we want this method to work perfectly.

Title and authors: Lu: They also pointed out that they need native CUDA kernel implementations for SALU and its gated variants if people want to actually use them in practice on hardware efficiently.

Meng: That’s something I'll keep in mind for implementation because the efficiency of the kernel matters a lot when you are running large models on real traffic.

Lalam: To wrap up, the paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" shows that replacing rigid statistical normalization with SALU and its variants gives us a way to handle stabilization through intrinsic geometry rather than external statistics.

Tom: It sounds like they’ve successfully shown that we can get competitive accuracy on CIFAR-one hundred without any explicit normalization layers at all, which is a pretty strong result.

Jane: They’ve also proven that this approach isn't just a trick for one architecture; it has been shown to work in Vision Transformers too, suggesting it could be more broadly useful.

Lu: The idea of total plasticity means we are learning how the signal propagates at every level, which is a big conceptual shift away from partitioning learnability across separate components.

Meng: For me, the practical implication is that if we can design an activation function that handles stabilization intrinsically without needing constant external tuning based on batch statistics, it makes model maintenance much less fragile.

Lalam: So to summarize the findings of "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," they introduce SALU for intrinsic signal stabilization through geometry-driven dynamics, and then build SWALU and GALU on top of it for learnable gating.

Tom: It seems like the main thing here is moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat idea.

Jane: It’s a neat idea because it connects stabilization directly to how we define our nonlinearities in neural networks, instead of treating them as separate tools.

Lu: The depth dependent stratification they observed is important because it shows that the network learns to use different geometric regimes for early and deep layers, which is a kind of internal structural learning.

Meng: So if we want to deploy this, we’re not just looking at one fixed activation; we’re looking at a system where the activation itself evolves its geometry during training based on the data it sees.

Lalam: We need to keep an eye on those limitations they mentioned about learning rate schedules and sample diversity, because that tells us when to expect performance to drop if we don't tune things right.

The paper's summary: Tom: So, to recap, SaluNet is proposing we ditch those traditional normalization layers like Batch Normalization and replace them with this single learnable activation function called SALU, which uses its own shape and scale to handle signal stabilization geometrically instead of relying on batch statistics.

Jane: That’s a big idea because it means the network learns how to stabilize itself intrinsically, rather than having an external layer do all the heavy lifting based on what's happening in the current batch.

Lu: Exactly. The paper introduces this concept of total plasticity where connections, stabilization, and gating dynamics are all learned through this geometry-driven process. It’s a unified approach to training deep networks that replaces those rigid statistical constraints with something neuron-level.

Meng: From my side at the startup, the main thing I’m interested in is how this affects deployment cost and stability at scale. If we can remove those external statistics, does that mean fewer moving parts in our inference pipeline?

Tom: It does; it means we remove those running means and variances you have to track constantly during training and inference. And the results on CIFAR-one hundred are pretty solid, hitting eighty-three point two five percent accuracy without any normalization layers.

Jane: That’s a high number for a model that doesn't even have a standard normalization layer in it, so what makes that result stand out to you?

Tom: What stands out is the robustness at batch size one. The paper shows SaluNet keeps strong performance even when you scale the batch size down to one, which is where most other normalized architectures start failing.

Lu: That’s because they’re showing that this learned geometry gives it a way to stay stable without needing those statistical buffers that collapse at small batches. And they analyze the learned geometry itself, and they find a depth dependent stratification.

Jane: Depth dependent? What does that mean for the network structure?

Lu: It means early layers tend to have strong saturation and narrow linear regimes, while deeper layers prioritize wide linear regimes and relaxed gating to maximize how much information they can hold. It’s like the network naturally structuring itself based on how deep it is.

Meng: So that means the model learns different geometric behaviors for different parts of its structure without us having to manually design those layer behaviors in advance? That sounds like a lot of flexibility, but I gotta ask, what are the caveats here?

Lu: The paper does flag some limitations; they say the framework can be sensitive to abrupt learning rate schedules and requires a short adaptation period for those intrinsic parameters 'a' and 'b' to readjust. They also need sufficient sample diversity for that geometric plasticity to converge well.

Tom: So it’s not a magic fix that works perfectly every single time; you still have to tune the learning rate schedule carefully and make sure your training data is diverse enough for the geometry to settle into its right shape.

Jane: That makes sense. It’s not just plugging in an activation function and hoping for the best; it’s about controlling that geometry through training dynamics, but you still have to manage the environment around it.

Lu: Yeah, and they also point out that for real-world hardware use, we still need native CUDA kernel implementations for SALU and its gated variants if we want to get the performance right on a GPU.

Meng: Right. So the idea is powerful conceptually—stabilization through geometry—but as an engineer, I’m looking at those implementation hurdles and the tuning requirements.

Tom: It really does seem like they’ve successfully moved the burden of stabilization from external statistical mechanisms into the activation function itself, which is a neat way to think about model stability.

Jane: It connects stabilization directly to how we define our nonlinearities in neural networks, instead of treating them as separate tools you bolt on later.

Lu: And that unified principle extends beyond CNNs; they showed SaluNet-T works well for Vision Transformers, suggesting this idea applies across different architectures under the same total plasticity concept.

Meng: If it scales and works across those different architectures, that makes it much more attractive for production environments where we have a mix of model types to maintain.

Tom: So the main thing here is moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat way to think about model stability.

Jane: It’s a neat idea because it connects stabilization directly to how we define our nonlinearities in neural networks, instead of treating them as separate tools you bolt on later.

Lu: And that unified principle extends beyond CNNs; they showed SaluNet-T works well for Vision Transformers, suggesting this idea applies across different architectures under the same total plasticity concept.

Meng: If it scales and works across those different architectures, that makes it much more attractive for production environments where we have a mix of model types to maintain.

The paper's improvements: Tom: So, we’re talking about how these improvements take SaluNet from just being an interesting concept to something practically useful in training models.

Jane: What are they suggesting is that by making the stabilization mechanism learnable through geometry, you unlock a whole new kind of control over the signal propagation inside the network.

Lu: They are showing that this approach allows for geometric plasticity, where the network learns its own optimal behavior by adjusting those shape and scale parameters dynamically during training.

Meng: I’m interested in what they suggest about gradient flow, because we always worry about optimization stability when you get rid of those standard normalization layers.

Tom: They suggest using a specific form of SALU where the derivative has a certain polynomial decay, like SALU prime equals SALU divided by x cubed times one over 'a squared', which helps preserve usable gradients over a wider dynamic range.

Jane: That’s important because fixed activations often cause those gradients to vanish or explode too quickly, and this mechanism keeps them in check better.

Lu: And they also discuss how the framework enables organic self normalization, where the final embedding layer naturally drives its output distributions toward a symmetric, quasi-Gaussian profile without needing an explicit normalization layer there.

Meng: That’s a big win for deployment because it means less overhead in our model architecture and we don't have to worry about that final layer's statistics during inference.

Tom: And they also point out the deep stratification again, meaning early layers handle saturation while deep layers use wider linear regimes for representation, which is how you get both strong detail and broad feature capture simultaneously.

Jane: That means the network learns a hierarchical way to process information based on its depth, which is really fascinating from a science communication standpoint.

Lu: It shifts the whole paradigm; instead of having external layers impose constraints on the signal, we are learning how the neuron itself responds and stabilizes its own output through these geometry-driven dynamics.

Meng: So if this works as advertised—no normalization, better stability at batch size one—does that mean we can cut down on other complex stabilizing components in our pipelines?

Tom: It means yes; it suggests that shifting the stabilization burden to intrinsic neuron level dynamics offers four fundamental advantages, including geometric plasticity and organic self normalization.

Jane: That's the core message: you get better performance and stability by learning how the signal propagates at every level, instead of just imposing a fixed statistical correction on top.

Lu: The implication is that we can design architectures where the stabilization mechanism is fully integrated into the activation function itself, unifying stabilization and gating dynamics into a single component.

Meng: So to summarize what they’re pushing: learnable geometry handles stabilization and gating without needing external batch statistics or affine parameters, leading to better stability at batch size one and better performance across different model types like Transformers.

Conclusion: Tom: So we’re wrapping up this look at "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers." The main idea is that by using SALU and its gated variants, the network learns how to stabilize itself geometrically instead of relying on external batch statistics.

Jane: That’s right. It shows that we can replace rigid statistical normalization with an intrinsic, learnable geometric control mechanism for signal propagation.

Lu: The paper concludes that this framework achieves state of the art performance on CIFAR-one hundred and even competitive accuracy on ImageNet-1K without any explicit normalization layers at all <ref:2606.02927#pg1>.

Meng: That’s a solid number, but I still see the caveat about needing sufficient sample diversity for that geometric plasticity to converge effectively, which means we can't just throw it into production and expect perfect results every time.

Tom: Exactly. It’s not just a trick; there are still tuning requirements like learning rate schedules and data diversity that need careful management for this to perform well across the board.

Jane: So what does this mean for the world of deep learning? It suggests that we can fundamentally shift how we think about model stability, moving away from imposed statistical corrections toward intrinsic neuron-level dynamics.

Lu: The implication is a more unified principle of total plasticity where connections, stabilization, and gating are all jointly learned through this geometry-driven process.

Meng: From an engineering standpoint at the startup level, if we can remove those external statistical layers without sacrificing performance on real traffic data, that really simplifies the entire deployment pipeline.

Tom: It means we can get better stability at batch size one and across different model types like Vision Transformers by applying this single unified principle of total plasticity.

Jane: It’s about moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat way to think about model stability.

Lu: The depth dependent stratification they observed is important because it shows that the network learns to use different geometric regimes for early and deep layers, which is a kind of internal structural learning.

Meng: So if we want to deploy this, we’re not just looking at one fixed activation; we’re looking at a system where the activation itself evolves its geometry during training based on the data it sees.

Tom: It seems like they’ve successfully shown that replacing rigid statistical normalization with SALU and its variants gives us a way to handle stabilization through intrinsic geometry rather than external statistics.

Jane: They’ve also proven that this approach isn't just a trick for one architecture; it has been shown to work in Vision Transformers too, suggesting it could be more broadly useful across diverse architectures.

Lu: The authors conclude that shifting the stabilization burden to intrinsic neuron level dynamics offers four fundamental advantages, including geometric plasticity and organic self normalization.

Meng: So to wrap up what we’ve heard on "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," it’s a powerful framework that removes external statistical constraints by learning stabilization through geometry.

Tom: It really does seem like they’ve successfully shown that we can get competitive accuracy on CIFAR-one hundred without any explicit normalization layers at all, which is a pretty strong result.

Jane: They’ve also proven that this approach isn't just a trick for one architecture; it has been shown to work in Vision Transformers too, suggesting it could be more broadly useful across diverse architectures.

Lu: The idea of total plasticity means we are learning how the signal propagates at every level, which is a big conceptual shift away from partitioning learnability across separate components.

Meng: For me, the practical implication is that if we can design an activation function that handles stabilization intrinsically without needing constant external tuning based on batch statistics, it makes model maintenance much less fragile.

Tom: So to summarize the findings of "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," they introduce SALU for intrinsic signal stabilization through geometry-driven dynamics, and then build SWALU and GALU on top of it for learnable gating.

Jane: It seems like the main thing here is moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat idea.

Lu: And that unified principle extends beyond CNNs; they showed SaluNet-T works well for Vision Transformers, suggesting this stabilization strategy generalizes across diverse architectures beyond convolutional networks.

Mouad Zaied

National Engineering School of Gabes · University of Gabes

cs.CV

Submitted: 2026-06-01

Updated: 2026-10-31

Comments: Revised version with clarified theoretical analysis, updated experimental results and protocols, and public code and pretrained checkpoints: https://github.com/MouradZaied/SaluNet/

Code: https://github.com/omihub777/ViT-CIFAR

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks <ref:2606.02927#pg2> This work demonstrates that they…

Key concepts

SALU Activation
SALU is a bounded, learnable activation function defined by SALU(x; a, b) = ax√1 + abx². It stabilizes signals intrinsically through its geometry. Its derivative's maximum at the origin provides explicit, learnable control over local gradient amplification.
Total Plasticity
This is the core principle where all signal propagation components—connections (weights), stabilization (SALU), and gating dynamics (SWALU/GALU)—become fully learnable. It replaces rigid statistical constraints with intrinsic neuron-level dynamics for effective deep learning.
Geometric Stratification
The learned geometry shows that early layers use strong saturation to dampen variance, while deep layers favor wide linear regimes and relaxed gating. This depth-dependent redistribution of geometric invariants demonstrates the network learns a continuous control mechanism over signal propagation.

Terminology

Summary

Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks <ref:2606. p. 0

Where the paper says it

zaied@univgb. tn June 3, 2026 ABSTRACT Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks.

02927#pg2> This work demonstrates that they can be fully replaced by a single learnable activation mechanism <ref:2606. p. 0
Where the paper says it

tn June 3, 2026 ABSTRACT Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks. This work demonstrates that they can be fully replaced by a single learnable activation mechanism.

02927#pg2> The paper introduces SALU, a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters <ref:2606. p. 0
Where the paper says it

We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters such as PReLU’s slope α rapidly lose meaningful adaptability when paired with normalization layers, which dominate the statistical dynamics of intermediate representations. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters.

02927#pg2> Building on SALU, the authors propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity <ref:2606. p. 0
Where the paper says it

Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU.

02927#pg2>.

The Core Concept of Total Plasticity

The central idea is that all major components of signal propagation become fully learnable through geometric plasticity <ref:2606.02927#pg2> This means connections are learned via network weights, stabilization is learned through SALU-based adaptive saturation, and gating dynamics are learned via SWALU and GALU <ref:2606. p. 0

Where the paper says it

Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU.

02927#pg2>. This framework aims to replace external statistical constraints with intrinsic neuron-level dynamics <ref:2606. p. 3
Where the paper says it

We identify the plasticity suppression effect induced by standard normalization layers, where learnable activation parameters such as PReLU’s slope α cease meaningful adaptation during training, revealing a fundamental misalignment between external stabilization and neuron-level learnability. We introduce SALU (Saturated Adaptive Linear Unit), a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters, and can directly replace normalization layers such as BatchNorm and LayerNorm.

02927#pg2>.

The SALU Activation Mechanism

SALU is defined as SALU(x; a, b) = ax√1 + abx2 where a > 0, b > 0 <ref:2606.02927#pg2> It acts as a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics <ref:2606. p. 0

Where the paper says it

We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters such as PReLU’s slope α rapidly lose meaningful adaptability when paired with normalization layers, which dominate the statistical dynamics of intermediate representations. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters.

02927#pg2>. The derivative of SALU is given by d/dx SALU(x; a, b) = a (1 + abx2)3 / 2 <ref:2606. p. 4
Where the paper says it

Complete derivations are provided in Appendix B. 1 Derivative and Gradient Propagation The derivative of SALU is: d dx SALU(x; a, b) = a (1 + abx2) 3/2 .

02927#pg2> This derivative reaches its maximum at the origin: SALU'(0) = a, providing explicit learnable control over local gradient amplification <ref:2606.02927#pg2>.

Unified Gating Mechanisms

To extend stabilization to modern architectures, the authors introduce SWALU and GALU as learnable gated activations <ref:2606.02927#pg2> SWALU is a SALU-based extension of Swish, where sigmoid-based gating is replaced by a SALU-based gate <ref:2606. p. 2

Where the paper says it

A PREPRINT Building upon SALU, we further introduce learnable variants of widely used gated activations: • SWALU: a SALU-based extension of Swish, where sigmoid-based gating is replaced by a SALU-based gate, making the gating behavior fully learnable. • GALU: a SALU-based generalization of GELU-style gating, designed for transformer and attention-based architectures, where SALU similarly drives the gating mechanism.

02927#pg2>. GALU is a SALU-based generalization of GELU-style gating, designed for transformer and attention-based architectures <ref:2606. p. 2
Where the paper says it

A PREPRINT Building upon SALU, we further introduce learnable variants of widely used gated activations: • SWALU: a SALU-based extension of Swish, where sigmoid-based gating is replaced by a SALU-based gate, making the gating behavior fully learnable. • GALU: a SALU-based generalization of GELU-style gating, designed for transformer and attention-based architectures, where SALU similarly drives the gating mechanism.

02927#pg2> These components collectively enable fully learnable stabilization and gating within a single unified principle <ref:2606. p. 3
Where the paper says it

We introduce SALU (Saturated Adaptive Linear Unit), a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters, and can directly replace normalization layers such as BatchNorm and LayerNorm. We propose SWALU and GALU, extending SALU-based geometric plasticity to gated mechanisms across convolutional and transformer architectures, enabling fully learnable stabilization and gating within a single unified principle.

02927#pg2>.

Empirical Validation and Robustness

The framework achieves state-of-the-art performance, notably SaluNet-C-18 reaching 97.35% on CIFAR10 and 83. p. 0

Where the paper says it

With a ResNet-18 backbone, SaluNet-C-18 achieves 97. 35% on CIFAR10 and 83.

25% on CIFAR-100 without any normalization layers <ref:2606. p. 0
Where the paper says it

35% on CIFAR10 and 83. 25% on CIFAR-100 without any normalization layers, and maintains 93.

p. 0
Where the paper says it

35% on CIFAR10 and 83. 25% on CIFAR-100 without any normalization layers, and maintains 93.

02927#pg2>. The model exhibits extreme robustness to batch size scaling, maintaining strong performance even at the unitary limit (BS = 1) where normalized architectures fail to converge <ref:2606. p. 2
Where the paper says it

25% best) on CIFAR-100, surpassing both BatchNorm-based baselines such as timm A2 [6] and AdAutoMixup [7], and normalization-free approaches such as NF-ResNet [8], without any normalization layers. • Extreme robustness to batch size scaling: At the unitary limit (BS = 1), SaluNet-C-18 maintains 93.

02927#pg2> Furthermore, SaluNet-C-50 achieves competitive accuracy on ImageNet-1K within a standard 90-epoch budget <ref:2606. p. 2
Where the paper says it

• State-of-the-art performance on ImageNet under short training budgets: SaluNet-C-50 achieves 78. 67% Top-1 accuracy on ImageNet-1K under the standard 224 × 224 setting within only 90 epochs, and reaches 79.

02927#pg2>.

Emergent Geometric Stratification

The analysis of the learned geometry reveals a depth-dependent stratification where early layers exhibit strong saturation and narrow linear regimes, while deep layers prioritize wide linear regimes and relaxed gating to maximize representational capacity <ref:2606.02927#pg2> This depth-dependent redistribution of geometric invariants demonstrates that SaluNet learns a continuous geometric control mechanism over signal propagation <ref:2606. p. 7

Where the paper says it

1 SWALU: Swish Adaptive Learnable Unit Swish is a smooth, non-monotonic activation function defined as: Swish(x) = x · σ(x), where σ(x) is the sigmoid function [14]. It can be interpreted as an input-dependent gating mechanism, where the signal x is modulated by a data-dependent gate σ(x) ∈ (0, 1), allowing the network to smoothly control information flow instead of applying a hard threshold.

02927#pg2>.

Biological Interpretation

SALU enables a form of intrinsic plasticity where neurons learn how to respond, not merely how strongly they connect <ref:2606.02927#pg2> This suggests that normalization layers actively suppress the total plasticity that enables deep networks to learn effectively <ref:2606. p. 0

Where the paper says it

23% at 288×288 resolution. These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively.

02927#pg2>. The authors conclude that shifting the stabilization burden from aggregate batch statistics to intrinsic neuron-level dynamics offers four fundamental advantages, including Geometric Plasticity and Organic Self-Normalization <ref:2606. p. 1
Where the paper says it

This raises a fundamental question: instead of relying on external, batch-dependent mechanisms that constrain neuron-level learnability, can we design activations that preserve learnable parameters such as α while intrinsically stabilizing signal propagation? We argue that such a design should satisfy four fundamental properties: (i) boundedness, to prevent uncontrolled signal growth; (ii) learnability, to preserve adaptive behavior; (iii) statelessness, to eliminate dependence on batch statistics; and (iv) self-contained stabilization, avoiding external affine normalization parameters such as γ and β.

p. 1
Where the paper says it

This raises a fundamental question: instead of relying on external, batch-dependent mechanisms that constrain neuron-level learnability, can we design activations that preserve learnable parameters such as α while intrinsically stabilizing signal propagation? We argue that such a design should satisfy four fundamental properties: (i) boundedness, to prevent uncontrolled signal growth; (ii) learnability, to preserve adaptive behavior; (iii) statelessness, to eliminate dependence on batch statistics; and (iv) self-contained stabilization, avoiding external affine normalization parameters such as γ and β.

02927#pg2>.

Limitations and Future Work

The framework shows sensitivity to abrupt learning rate schedules, requiring a short adaptation period for the intrinsic parameters (a, b) to re-adjust <ref:2606.02927#pg2> Additionally, the joint optimization of weight and activation parameters demands sufficient sample diversity for geometric plasticity to converge effectively <ref:2606. p. 3

Where the paper says it

10% on CIFAR-100, demonstrating that learnable activation geometry extends effectively beyond convolutional networks. • Improved sample efficiency: SaluNet models reach peak performance earlier in training, indicating faster convergence dynamics despite the complete absence of explicit normalization layers.

02927#pg2>. Future work will focus on native CUDA kernel implementations for SALU and its gated variants <ref:2606. p. 2
Where the paper says it

A PREPRINT Building upon SALU, we further introduce learnable variants of widely used gated activations: • SWALU: a SALU-based extension of Swish, where sigmoid-based gating is replaced by a SALU-based gate, making the gating behavior fully learnable. • GALU: a SALU-based generalization of GELU-style gating, designed for transformer and attention-based architectures, where SALU similarly drives the gating mechanism.

02927#pg2>.

Related Work Context

SaluNet differs from prior work by making stabilization itself fully learnable through geometric plasticity, extending this learnable geometry to gating mechanisms via SWALU and GALU <ref:2606.02927#pg2> This unifies stabilization and activation into a single learnable component, realizing the principle of total plasticity within a single unified mechanism <ref:2606. p. 3

Where the paper says it

We propose SWALU and GALU, extending SALU-based geometric plasticity to gated mechanisms across convolutional and transformer architectures, enabling fully learnable stabilization and gating within a single unified principle. We introduce SaluNet, a new paradigm for deep learning grounded in total plasticity, in which connections, stabilization, and gating dynamics are all fully learnable.

02927#pg2>. The authors position their work at the intersection of explicit normalization and parametric activations <ref:2606. p. 6
Where the paper says it

4 Conceptual Contribution SALU challenges the prevailing assumption that stable deep learning requires explicit signal normalization. Instead, it demonstrates that a carefully designed parametric activation can stabilize information flow through its intrinsic geometry alone.

02927#pg2>.

Conclusion Summary

In this paper, SaluNet, a normalization-free framework grounded in the principle of total plasticity, in which all major components of signal propagation — connections, stabilization, and gating — are jointly learnable <ref:2606.02927#pg2> By replacing rigid statistical normalization with the bounded, adaptive geometry of SALU and its gated variants SWALU and GALU, we demonstrate that deep neural networks can achieve competitive performance without any explicit normalization layers <ref:2606. p. 3

Where the paper says it

We introduce SALU (Saturated Adaptive Linear Unit), a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters, and can directly replace normalization layers such as BatchNorm and LayerNorm. We propose SWALU and GALU, extending SALU-based geometric plasticity to gated mechanisms across convolutional and transformer architectures, enabling fully learnable stabilization and gating within a single unified principle.

02927#pg2>. The successful integration into Vision Transformers (SaluNet-T) further suggests that this stabilization strategy generalizes across diverse architectures beyond convolutional networks <ref:2606. p. 3
Where the paper says it

While both demonstrate strong performance, they are evaluated exclusively on Transformer architectures. SaluNet differs in three fundamental ways: (i) it eliminates normalization not by substituting a fixed functional form, but by making stabilization itself fully learnable through geometric plasticity; (ii) it extends learnable geometry to gating mechanisms via SWALU and GALU; and (iii) it applies seamlessly to both convolutional networks and Vision Transformers under a single unified principle of total plasticity.

02927#pg2>.

References

[1] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. p. 0

Where the paper says it

Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU.

In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 448–456, 2015 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. p. 0

Where the paper says it

zaied@univgb. tn June 3, 2026 ABSTRACT Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks.

arXiv preprint arXiv:1607. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

06450, 2016 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[3] Vinod Nair and Geoffrey Hinton. Rectified linear units improve restricted boltzmann machines. p. 0

Where the paper says it

We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters such as PReLU’s slope α rapidly lose meaningful adaptability when paired with normalization layers, which dominate the statistical dynamics of intermediate representations. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters.

In Proceedings of the International Conference on Machine Learning (ICML), 2010 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[4] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELU). p. 7

Where the paper says it

Replacing tanh with SALU gives the Swish Adaptive Learnable Unit (SWALU): SWALU(x) = x 2 (1 + SALU(x; a, b)) (9) With initialization a = b = 1, SWALU behaves similarly to Swish, while remaining fully learnable throughout training. 2 GALU: Gaussian Adaptive Learnable Unit GELU employs a smooth Gaussian-inspired gate [4], commonly approximated as: GELU(x) ≈ x 2 1 + tanh r 2 π (x + 0.

arXiv preprint arXiv:1606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

08415, 2016 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. p. 2

Where the paper says it

67% Top-1 accuracy on ImageNet-1K under the standard 224 × 224 setting within only 90 epochs, and reaches 79. 23% at 288 × 288 resolution, surpassing both BatchNorm-based baselines including ResNet Strikes Back A3 [6] trained for 100 epochs, and normalization-free approaches such as NF-ResNet-50 [9], without any normalization layers.

In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026–1034, 2015 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[6] Ross Wightman, Hugo Touvron, and Hervé Jégou. ResNet strikes back: An improved training procedure in timm. p. 2

Where the paper says it

67% Top-1 accuracy on ImageNet-1K under the standard 224 × 224 setting within only 90 epochs, and reaches 79. 23% at 288 × 288 resolution, surpassing both BatchNorm-based baselines including ResNet Strikes Back A3 [6] trained for 100 epochs, and normalization-free approaches such as NF-ResNet-50 [9], without any normalization layers.

arXiv preprint arXiv:2110. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

00476, 2021 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[7] X. Zhu et al. Adversarial AutoMixup. arXiv preprint arXiv:2312. p. 0

Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

11954, 2024 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[8] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. p. 3

Where the paper says it

This mirrors the intrinsic adaptability of biological neurons, which achieve stable signal propagation without any external statistical constraint. We validate SaluNet across diverse architectures and datasets, establishing state-of-the-art performance among ResNet-18 models on CIFAR, competitive large-scale performance on ImageNet under short training budgets, and remarkable robustness at batch size 1, a regime where normalized architectures fail to converge.

In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 1000–1010, 2021 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[9] Andrew Brock, Soham De, and Samuel L. Smith. Characterizing signal propagation to close the performance gap in unnormalized ResNets. p. 3

Where the paper says it

This mirrors the intrinsic adaptability of biological neurons, which achieve stable signal propagation without any external statistical constraint. We validate SaluNet across diverse architectures and datasets, establishing state-of-the-art performance among ResNet-18 models on CIFAR, competitive large-scale performance on ImageNet under short training budgets, and remarkable robustness at batch size 1, a regime where normalized architectures fail to converge.

International Conference on Learning Representations (ICLR), 2021 <ref:2606. p. 2
Where the paper says it

This is made possible by geometric plasticity: because SALU, SWALU, and GALU all adapt their shape, gain, and saturation during training, stabilization and gating are no longer fixed operations imposed on the network, they become intrinsic degrees of freedom that co-evolve with the learned representations. The resulting network behaves as a fully adaptive dynamical system in which representation learning, gating, and stabilization co-evolve during training, without any external statistical constraint.

02927#pg2>

[10] Biao Zhang and Rico Sennrich. Root mean square layer normalization. p. 8

Where the paper says it

Since SALU constitutes the fundamental stabilization mechanism of the proposed framework, we first analyze its dynamical properties in isolation before extending the discussion to the complete SALU-based residual and transformer architectures. 1 Forward Signal Propagation Consider a deep network where each layer computes h ℓ+1 = Wℓϕ(h ℓ ; aℓ, bℓ), (16) where ϕ denotes the SALU activation and Wℓ ij ∼ N (0, σ2 w/n) under standard mean-field assumptions.

In Advances in Neural Information Processing Systems (NeurIPS), 2019 <ref:2606.02927#pg2>

[11] Yuxin Wu and Kaiming He. Group normalization. p. 3

Where the paper says it

Relation to Prior Work. Normalization layers such as Batch Normalization [1], Layer Normalization [2], RMSNorm [10], and Group Normalization [11] have long been considered essential to stable training, but impose fixed statistical constraints that suppress neuron-level learnability.

In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19, 2018 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[12] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization, 2025 <ref:2606. p. 0

Where the paper says it

Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU. In this framework, SALU directly replaces normalization layers such as BatchNorm and LayerNorm, while SWALU and GALU replace standard activation functions such as ReLU, Swish, and GELU, covering both convolutional networks (CNNs) and Vision Transformers (ViTs) without architectural compromise.

02927#pg2>

[13] Mingzhi Chen, Taiming Lu, Jiachen Zhu, Mingjie Sun, and Zhuang Liu. Stronger normalization-free transformers, 2025 <ref:2606. p. 0

Where the paper says it

These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively. Keywords normalization-free deep learning · learnable activation functions · total plasticity · SALU · SWALU · GALU · convolutional neural networks · vision transformers · signal stabilization · batch size robustness ∗ https://enig.

02927#pg2>

[14] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. p. 0

Where the paper says it

Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU. In this framework, SALU directly replaces normalization layers such as BatchNorm and LayerNorm, while SWALU and GALU replace standard activation functions such as ReLU, Swish, and GELU, covering both convolutional networks (CNNs) and Vision Transformers (ViTs) without architectural compromise.

arXiv preprint arXiv:1710. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

05941, 2017 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[15] Omihub777. Vision transformer (ViT) implementation for CIFAR-10 and CIFAR-100. p. 3

Where the paper says it

We validate SaluNet across diverse architectures and datasets, establishing state-of-the-art performance among ResNet-18 models on CIFAR, competitive large-scale performance on ImageNet under short training budgets, and remarkable robustness at batch size 1, a regime where normalized architectures fail to converge. We further demonstrate that SaluNet extends seamlessly to Vision Transformers, where SaluNet-T-CIFAR outperforms the standard LayerNorm-GELU configuration on both CIFAR-10 and CIFAR-100.

https://github. p. 0
Where the paper says it

These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively. Keywords normalization-free deep learning · learnable activation functions · total plasticity · SALU · SWALU · GALU · convolutional neural networks · vision transformers · signal stabilization · batch size robustness ∗ https://enig.

com/omihub777/ViT-CIFAR, 2021 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[16] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. p. 3

Where the paper says it

This mirrors the intrinsic adaptability of biological neurons, which achieve stable signal propagation without any external statistical constraint. We validate SaluNet across diverse architectures and datasets, establishing state-of-the-art performance among ResNet-18 models on CIFAR, competitive large-scale performance on ImageNet under short training budgets, and remarkable robustness at batch size 1, a regime where normalized architectures fail to converge.

arXiv preprint arXiv:1706. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02677, 2017 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[17] Takuya Akiba, Shuji Suzuki, Keisuke Fukuda, Satoshi Kobayashi, Yuichi Suzuki, and Kohei Komuro. Extremely large minibatch SGD: Training ResNet-50 on ImageNet in 15 minutes. p. 3

Where the paper says it

This mirrors the intrinsic adaptability of biological neurons, which achieve stable signal propagation without any external statistical constraint. We validate SaluNet across diverse architectures and datasets, establishing state-of-the-art performance among ResNet-18 models on CIFAR, competitive large-scale performance on ImageNet under short training budgets, and remarkable robustness at batch size 1, a regime where normalized architectures fail to converge.

arXiv preprint arXiv:1711. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

04325, 2017 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. p. 8

Where the paper says it

We analyze how these properties influence signal and gradient propagation in deep neural networks. Since SALU constitutes the fundamental stabilization mechanism of the proposed framework, we first analyze its dynamical properties in isolation before extending the discussion to the complete SALU-based residual and transformer architectures.

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016 <ref:2606.02927#pg2>

[19] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. p. 1

Where the paper says it

This suggests that stability and plasticity are not inherently incompatible, but their interaction is fundamentally misaligned in standard architectures. This motivates a shift in perspective: effective stabilization should emerge from a mechanism that enables proper learning of activation parameters, rather than from external normalization.

In European Signal Processing Conference (EUSIPCO), pages 606–610, 2007 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[20] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. p. 1

Where the paper says it

In contrast, SALU maintains stable yet adaptive dynamics, preventing both collapse and uncontrolled drift. 1 Introduction Deep neural networks owe much of their success to two fundamental components: normalization layers that stabilize optimization dynamics [1, 2], and activation functions that introduce nonlinearity and expressive power [3, 4].

Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020 <ref:2606. p. 0
Where the paper says it

SALUNET: ENABLING TOTAL PLASTICITY IN NORMALIZATION-FREE DEEP NETWORKS A PREPRINT Mouad Zaied* Department of electrical engineering National Engineering School of Gabes (ENIG) University of Gabes Avenue Omar Ibn El Khattab, Zrig Eddakhlania 6029, Gabes, Tunisia mourad. zaied@univgb.

02927#pg2>

[21] Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. p. 0

Where the paper says it

These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively. Keywords normalization-free deep learning · learnable activation functions · total plasticity · SALU · SWALU · GALU · convolutional neural networks · vision transformers · signal stabilization · batch size robustness ∗ https://enig.

In Advances in Neural Information Processing Systems (NeurIPS), pages 901–909, 2016 <ref:2606.02927#pg2>

[22] Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch dependence in the training of deep neural networks. p. 0

Where the paper says it

zaied@univgb. tn June 3, 2026 ABSTRACT Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks.

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. p. 2

Where the paper says it

A PREPRINT Building upon SALU, we further introduce learnable variants of widely used gated activations: • SWALU: a SALU-based extension of Swish, where sigmoid-based gating is replaced by a SALU-based gate, making the gating behavior fully learnable. • GALU: a SALU-based generalization of GELU-style gating, designed for transformer and attention-based architectures, where SALU similarly drives the gating mechanism.

Advances in Neural Information Processing Systems (NeurIPS), pages 5998–6008 <ref:2606.02927#pg2>

[24] David Balduzzi, Marcus Frean, Lennox Leary, J. P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If ResNets are the answer, then what is the question? p. 1

Where the paper says it

From Statistics to Geometry. This raises a fundamental question: instead of relying on external, batch-dependent mechanisms that constrain neuron-level learnability, can we design activations that preserve learnable parameters such as α while intrinsically stabilizing signal propagation?

In Proceedings of the 34th International Conference on Machine Learning, pages 342–350, 2017 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[25] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. p. 0

Where the paper says it

Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU.

International Conference on Learning Representations (ICLR), 2019 <ref:2606. p. 2
Where the paper says it

This is made possible by geometric plasticity: because SALU, SWALU, and GALU all adapt their shape, gain, and saturation during training, stabilization and gating are no longer fixed operations imposed on the network, they become intrinsic degrees of freedom that co-evolve with the learned representations. The resulting network behaves as a fully adaptive dynamical system in which representation learning, gating, and stabilization co-evolve during training, without any external statistical constraint.

02927#pg2>

[26] Shibani Santurkar, Dimitri P. Bertsekas, Nikunj Saunshi, and Kannan Ramchandran. How does batch normalization help optimization? p. 1

Where the paper says it

In contrast, SALU maintains stable yet adaptive dynamics, preventing both collapse and uncontrolled drift. 1 Introduction Deep neural networks owe much of their success to two fundamental components: normalization layers that stabilize optimization dynamics [1, 2], and activation functions that introduce nonlinearity and expressive power [3, 4].

Advances in Neural Information Processing Systems (NeurIPS), pages 2483–2493, 2018 <ref:2606.02927#pg2>

[27] Yunhao Ni, Yuxin Guo, Junlong Jia, and Lei Huang. On the nonlinearity of layer normalization. p. 1

Where the paper says it

In contrast, SALU maintains stable yet adaptive dynamics, preventing both collapse and uncontrolled drift. 1 Introduction Deep neural networks owe much of their success to two fundamental components: normalization layers that stabilize optimization dynamics [1, 2], and activation functions that introduce nonlinearity and expressive power [3, 4].

In Proceedings of the International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research (PMLR), pages 37957–37998, 2024 <ref:2606.02927#pg2>

[31] Bin Liu, Yihui He, Hao Li, Guangrun Wang, Xiaodong Li, and Jianping Shi. SReLU: SeLU-style rectified linear unit activations. p. 0

Where the paper says it

Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), SALU(x; a, b) = ax √ 1 + abx2 , a > 0, b > 0 (1) a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU.

In Proceedings of the European Conference on Computer Vision (ECCV), pages 841–856, 2018 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[32] Luke Rood, Vincent van der Sar, et al. Zorro: Shape-controlled parametric activations. p. 7

Where the paper says it

Figure 4 illustrates these geometric regimes for SWALU and GALU under different values of (a, b). This theoretical flexibility suggests that SALU-based activations act as layer-specific geometric operators, capable of spanning a wide range of nonlinear behaviors within a single unified parametric family.

arXiv preprint arXiv:2403. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

12345, 2024 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

[33] Benjamin Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. p. 0

Where the paper says it

These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively. Keywords normalization-free deep learning · learnable activation functions · total plasticity · SALU · SWALU · GALU · convolutional neural networks · vision transformers · signal stabilization · batch size robustness ∗ https://enig.

Advances in Neural Information Processing Systems (NeurIPS), pages 3360–3368 <ref:2606.02927#pg2>

[34] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. p. 6

Where the paper says it

4 Conceptual Contribution SALU challenges the prevailing assumption that stable deep learning requires explicit signal normalization. Instead, it demonstrates that a carefully designed parametric activation can stabilize information flow through its intrinsic geometry alone.

International Conference on Learning Representations (ICLR), 2017 <ref:2606. p. 2
Where the paper says it

This is made possible by geometric plasticity: because SALU, SWALU, and GALU all adapt their shape, gain, and saturation during training, stabilization and gating are no longer fixed operations imposed on the network, they become intrinsic degrees of freedom that co-evolve with the learned representations. The resulting network behaves as a fully adaptive dynamical system in which representation learning, gating, and stabilization co-evolve during training, without any external statistical constraint.

02927#pg2>

[35] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Hendrik Jacobsen. On the invariance, stability and consistency of deep neural network representations. p. 0

Where the paper says it

These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively. Keywords normalization-free deep learning · learnable activation functions · total plasticity · SALU · SWALU · GALU · convolutional neural networks · vision transformers · signal stabilization · batch size robustness ∗ https://enig.

Advances in Neural Information Processing Systems (NeurIPS), pages 4505–4515 <ref:2606.02927#pg2>

[36] John Moody. Fast learning in networks of locally-tuned processing units. p. 0

Where the paper says it

Building on SALU, we propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity: rather than distributing learnability between restrictive normalization constraints and partially adaptive activations, SaluNet makes every component of signal propagation fully learnable—connections via network weights, stabilization via SALU-based adaptive saturation, and gating dynamics via SWALU and GALU. In this framework, SALU directly replaces normalization layers such as BatchNorm and LayerNorm, while SWALU and GALU replace standard activation functions such as ReLU, Swish, and GELU, covering both convolutional networks (CNNs) and Vision Transformers (ViTs) without architectural compromise.

In Advances in Neural Information Processing Systems (NeurIPS), pages 131–139, 1989 <ref:2606.02927#pg2>

[37] Ling Zhang and Paul B. Luh. Wavelet neural networks for function learning. p. 0

Where the paper says it

These results suggest that normalization layers, far from being necessary, actually suppress total plasticity, a property that biological neurons inherently possess, which enables deep networks to learn effectively. Keywords normalization-free deep learning · learnable activation functions · total plasticity · SALU · SWALU · GALU · convolutional neural networks · vision transformers · signal stabilization · batch size robustness ∗ https://enig.

IEEE Transactions on Signal Processing, 43(6):1485–1497, 1995 <ref:2606. p. 0
Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927#pg2>

Conclusion

In this paper, SaluNet reaches 97.35% on CIFAR-10 and 83. p. 0

Where the paper says it

35% on CIFAR10 and 83. 25% on CIFAR-100 without any normalization layers, and maintains 93.

25% on CIFAR-100 without any normalization layers <ref:2606.02927#pg2> The successful integration into Vision Transformers (SaluNet-T) further suggests that this stabilization strategy generalizes across diverse architectures beyond convolutional networks <ref:2606. p. 3
Where the paper says it

While both demonstrate strong performance, they are evaluated exclusively on Transformer architectures. SaluNet differs in three fundamental ways: (i) it eliminates normalization not by substituting a fixed functional form, but by making stabilization itself fully learnable through geometric plasticity; (ii) it extends learnable geometry to gating mechanisms via SWALU and GALU; and (iii) it applies seamlessly to both convolutional networks and Vision Transformers under a single unified principle of total plasticity.

02927#pg2>. The authors conclude that shifting the stabilization burden from aggregate batch statistics to intrinsic neuron-level dynamics offers four fundamental advantages, including Geometric Plasticity and Organic Self-Normalization <ref:2606.02927#pg2>.

Acknowledgements

This research was supported by the Google Cloud Research Credits program (Project: GCP Research Credits Project, Project ID: gcp-research-credits-project, Project Number: 695684110946) <ref:2606.02927#pg2> Compute resources were provided through Google Cloud Platform and were essential for conducting the large-scale experiments reported in this work <ref:2606.02927#pg2>. The paper is a preprint from arXiv: 2606. p. 0

Where the paper says it

tn/useruploads/files/CV-Pr-Mourad-ZAIED. pdf arXiv:2606.

02927v1 [cs. p. 0
Where the paper says it

pdf arXiv:2606. 02927v1 [cs.

CV] 1 Jun 2026 <ref:2606. p. 0
Where the paper says it

02927v1 [cs. CV] 1 Jun 2026

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Geometric Plasticity in Signal Propagation: The improved AI system will exhibit depth-dependent geometric stratification where early layers utilize strong saturation for variance damping while deep layers prioritize wide linear regimes for feature disentanglement, as seen in SaluNet-C-18's learned geometry analysis.

  2. Batch Size Independence: The system will maintain stable performance at the unitary batch limit (BS=1) without catastrophic failure, as SaluNet maintains strong performance throughout across all tested batch sizes due to its strict pointwise nature and lack of reliance on batch-level statistics.

  3. Organic Self-Normalization: The improved AI will naturally drive output distributions toward a symmetric, quasi-Gaussian profile at the final embedding layer, achieving this without any explicit normalization layers, which facilitates linear separability at the classifier input space.

  4. Enhanced Gradient Flow: By employing SALU's polynomial gradient decay where SALU′(x) = SALU(x) / x cubed · 1/a squared, the system will preserve usable gradients over a wider dynamic range compared to fixed activations, improving optimization stability in normalization-free networks.

  5. Architecture Agnosticism: The system can be seamlessly applied across both convolutional and transformer architectures by utilizing SaluNet-C for CNNs and SaluNet-T for Vision Transformers under a single unified principle of total plasticity.

  6. Learnable Gating Mechanisms: The improved AI will possess fully learnable gating dynamics via SWALU and GALU, allowing the network to adapt its shape, gain, and saturation during training without relying on external statistical constraints.

Abstract

Normalization stabilizes deep networks by adapting intermediate representations through prescribed centering and scaling rules. We investigate the complementary principle: leave representations in their original coordinates and adapt the geometry of the pointwise operator instead. Motivated by controlled evidence that Batch Normalization changes the learning dynamics of parametric activations, we introduce the Saturated Adaptive Linear Unit (SALU), a smooth bounded operator with two learned scalars per layer. A bijective reparameterization exposes its saturation amplitude S, transition width W, and central gain S/W. These coordinates control the bounded forward amplitude and distribute backward sensitivity over the input domain. In a homogeneous mean-field model, forward variance and backward gradient propagation share the normalized coordinate v/W squared, where v denotes the activation variance, while obeying distinct stability conditions. We reuse the same primitive to construct the SWALU and GALU adaptive gates and define SaluNet by replacing BN or LN with SALU and conventional activations with SALU-based gates. On CIFAR-100, controlled ResNet-18 experiments show that the complete SALU+SWALU construction outperforms its isolated substitutions. Additional experiments demonstrate trainability down to batch size one, while checkpoint-level representation analysis reveals a broader late-stage spectrum than in the matched BN+ReLU model. ImageNet-1K experiments further demonstrate scalability to ResNet-50 and ViT-B/16, including ViT-B/16 training with an effective batch size of 128. Together, these results support learnable pointwise geometry as a common statistics-free mechanism for signal stabilization and adaptive gating across convolutional networks and vision Transformers.

Sources

Related papers