Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers
summary
The gist
Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks <ref:2606.02927#pg2> This work demonstrates that they…
In short
SaluNet replaces traditional normalization layers with a single learnable activation, SALU, based on geometric dynamics. This framework introduces 'total plasticity,' where connections, stabilization, and gating are all learned through intrinsic neuron-level rules rather than external batch statistics. This allows deep networks to achieve state-of-the-art performance without any explicit normalization.
Key concepts
- SALU Activation
- SALU is a bounded, learnable activation function defined by SALU(x; a, b) = ax√1 + abx². It stabilizes signals intrinsically through its geometry. Its derivative's maximum at the origin provides explicit, learnable control over local gradient amplification.
- Total Plasticity
- This is the core principle where all signal propagation components—connections (weights), stabilization (SALU), and gating dynamics (SWALU/GALU)—become fully learnable. It replaces rigid statistical constraints with intrinsic neuron-level dynamics for effective deep learning.
- Geometric Stratification
- The learned geometry shows that early layers use strong saturation to dampen variance, while deep layers favor wide linear regimes and relaxed gating. This depth-dependent redistribution of geometric invariants demonstrates the network learns a continuous control mechanism over signal propagation.
Terminology used across episodes
This episode discusses
- Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers · Paper Radio
- Layer Normalization
- Gaussian Error Linear Units (GELUs)
- ResNet strikes back: An improved training procedure in timm
- Adversarial AutoMixup
- Searching for Activation Functions
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Performance Portable Monte Carlo Particle Transport on Intel, NVIDIA, and AMD GPUs
The paper
Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers · Read on arXiv
Mouad Zaied
National Engineering School of Gabes · University of Gabes
Normalization stabilizes deep networks by adapting intermediate representations through prescribed centering and scaling rules. We investigate the complementary principle: leave representations in their original coordinates and adapt the geometry of the pointwise operator instead. Motivated by controlled evidence that Batch Normalization changes the learning dynamics of parametric activations, we introduce the Saturated Adaptive Linear Unit (SALU), a smooth bounded operator with two learned scalars per layer. A bijective reparameterization exposes its saturation amplitude S, transition width W, and central gain S/W. These coordinates control the bounded forward amplitude and distribute backward sensitivity over the input domain. In a homogeneous mean-field model, forward variance and backward gradient propagation share the normalized coordinate v/W squared, where v denotes the activation variance, while obeying distinct stability conditions. We reuse the same primitive to construct the SWALU and GALU adaptive gates and define SaluNet by replacing BN or LN with SALU and conventional activations with SALU-based gates. On CIFAR-100, controlled ResNet-18 experiments show that the complete SALU+SWALU construction outperforms its isolated substitutions. Additional experiments demonstrate trainability down to batch size one, while checkpoint-level representation analysis reveals a broader late-stage spectrum than in the matched BN+ReLU model. ImageNet-1K experiments further demonstrate scalability to ResNet-50 and ViT-B/16, including ViT-B/16 training with an effective batch size of 128. Together, these results support learnable pointwise geometry as a common statistics-free mechanism for signal stabilization and adaptive gating across convolutional networks and vision Transformers.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Geometry, Not Statistics".
Jane: Here's the gist of the paper. Normalization layers such as Batch Normalization and Layer Normalization have long been considered essential to stable training in deep networks <ref:2606.02927#pg2> This work demonstrates that they can be fully replaced by a single learnable activation mechanism <ref:2606.02927#pg2> The paper introduces SALU, a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics, without relying on batch statistics or external affine parameters <ref:2606.02927#pg2> Building on SALU, the authors propose SaluNet, a new paradigm for deep learning grounded in the principle of total plasticity <ref:2606.02927#pg2>.
Tom: The Core Concept of Total Plasticity The central idea is that all major components of signal propagation become fully learnable through geometric plasticity <ref:2606.02927#pg2> This means connections are learned via network weights, stabilization is learned through SALU-based adaptive saturation, and gating dynamics are learned via SWALU and GALU <ref:2606.02927#pg2>.
Jane: This framework aims to replace external statistical constraints with intrinsic neuron-level dynamics <ref:2606.02927#pg2>.
Tom: The SALU Activation Mechanism SALU is defined as SALU(x; a, b) = ax√1 + abx2 where a > 0, b > 0 <ref:2606.02927#pg2> It acts as a bounded, learnable activation that provides intrinsic signal stabilization through geometry-driven dynamics <ref:2606.02927#pg2>.
Title and authors: Tom: So we're looking at this paper titled "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," and what it’s doing is replacing those big normalization layers like Batch Normalization with a single learnable activation mechanism.
Jane: That sounds complicated. Can you break down what the authors are actually proposing without all the math?
Lu: Basically, they say that instead of using things like Batch Normalization to stabilize training by looking at batch statistics, we can use something called SALU which is a bounded activation function that handles stabilization internally through its own shape and scale.
Meng: So it’s trying to get rid of the need for those external statistical constraints, which means no more needing batch statistics or extra affine parameters during training? Because from my point of view, relying on batch stats is just adding more dependencies that break when you move to a different setup.
Lalam: The core idea they are pushing is called total plasticity, where everything—the connections and the stabilization and the gating dynamics—becomes fully learnable through this geometric approach.
Tom: That’s a big claim. So what exactly is this SALU function they introduce? What does it look like mathematically?
Jane: The paper defines it as SALU of x with a and b, which is written as a times x root one plus a times b times x squared, where 'a' and 'b' are positive numbers.
Lu: That specific formula, SALU(x; a, b) = ax root one plus abxtwo with 'a' greater than zero and 'b' greater than zero, is what provides that bounded activation function they are using to stabilize the signal geometry <ref:2606.02927#pg1,SALU(x; a, b) = ax>.
Meng: And they claim this approach avoids relying on batch statistics or external affine parameters, so it’s not adding any new statistical overhead during the forward pass? That would be a huge win for deployment because we don't have to track those running means or variances.
Lalam: The paper shows that when you use standard normalization layers, like Batch Normalization, the learnable parameters in activations like PReLU rapidly lose their meaningful adaptability after about ten epochs on CIFAR-ten data because the normalization dominates the statistical dynamics there <ref:2606.02927#pg2>.
Tom: Wait, so they show that this plasticity suppression effect happens when you pair those learnable activation parameters with normalization layers, which is what makes them lose adaptability quickly?
Jane: Exactly. The authors point out that without Batch Normalization, the learnable slope in PReLU keeps drifting significantly throughout training because it’s highly adaptive but unstable.
Lu: In contrast, when you use SALU for stabilization instead of Batch Normalization, the geometry of the activation stays stable yet adaptive, which is what they call geometric plasticity.
Title and authors: Meng: So they are showing that the learning process can be controlled by how the activation function's shape and saturation change dynamically during training instead of relying on external layers to do that work.
Lalam: They build on SALU by introducing two more gated versions, SWALU and GALU, which are basically SALU-based extensions of Swish and GELU styles, making the gating behavior fully learnable too.
Tom: SWALU is a SALU extension of Swish where they replace the sigmoid-based gating with a SALU-based gate, meaning the gate itself adapts its shape and gain during training.
Jane: And GALU is a generalization of GELU-style gating specifically designed for transformer and attention based architectures, which means it’s tailored for those complex models.
Lu: The authors argue that SALU does more than just stabilize the signal; it acts as a learnable gate whose shape, gain, and saturation all adapt during training through geometric plasticity.
Meng: That sounds like a lot of things being learned at once—the stabilization, the gating dynamics, and the non-linear modulation—all emerging from that single bounded adaptive principle.
Lalam: They frame this as total plasticity where weights are learnable, and stabilization and gating are also learned through this mechanism instead of being imposed externally.
Tom: So what’s the practical takeaway here for someone building a deep network? Are we supposed to ditch all normalization layers entirely just because we can replace them with SALU?
Jane: Well, they show that SaluNet, which uses this framework, achieves state of the art performance on CIFAR-ten and CIFAR-one hundred without any explicit normalization layers.
Lu: They report that SaluNet-C-eighteen gets ninety-seven point three five percent accuracy on CIFAR-ten and eighty-three point two five percent on CIFAR-one hundred and SaluNet also hits competitive accuracy on ImageNet-1K within a standard ninety epoch budget, which is quite impressive.
Meng: That performance is good, but I’m curious about how it handles things at scale. Does this work maintain strong performance even when we test it at the unitary batch limit, meaning batch size one?
Lalam: Yes, the system exhibits extreme robustness to batch size scaling and maintains strong performance even at the unitary limit where normalized architectures usually fail to converge.
Tom: That addresses a major pain point for me because training stability often collapses when you drop the batch size down to one. So this method seems designed specifically to keep things stable under those tough conditions.
Jane: It suggests that instead of relying on batch statistics to smooth out the learning process, we can rely on intrinsic neuron level dynamics to achieve that stability.
Title and authors: Lu: They also found something interesting when they analyzed the learned geometry: there’s a depth dependent stratification where early layers show strong saturation and narrow linear regimes, but deep layers prioritize wide linear regimes and relaxed gating for more representational capacity.
Meng: So it means the network naturally learns how to behave differently in its different parts based on their depth, which is a very interesting way for the AI to structure itself without us having to manually design those layer behaviors.
Lalam: They conclude that this shift from aggregate batch statistics to intrinsic neuron level dynamics offers four fundamental advantages, including geometric plasticity and organic self normalization.
Tom: So we’re talking about Geometric Plasticity and Organic Self-Normalization as the main benefits derived from this SaluNet framework?
Jane: That’s right. It suggests that normalization layers might actually be suppressing the total plasticity that lets deep networks learn effectively in the first place, which is a pretty big observation.
Lu: The authors suggest that shifting this stabilization burden to intrinsic neuron level dynamics gives us geometric plasticity and organic self normalization as key gains for training deep networks without external constraints.
Meng: From a deployment standpoint, if we can achieve that kind of inherent stability without needing those complex statistical layers, it simplifies the entire pipeline significantly.
Lalam: The paper shows that SaluNet is also applicable across different architectures, like Vision Transformers, suggesting this stabilization strategy generalizes beyond just standard convolutional networks.
Tom: So if I’m looking at a CNN or a transformer architecture, I can use this single principle of total plasticity to handle both the stabilization and the gating mechanisms?
Jane: That’s the point; it unifies stabilization and activation into one component, meaning you can use SWALU for Swish-like models or GALU for transformer models under the same framework.
Lu: The paper demonstrates that SaluNet-T, which is applied to Vision Transformers, shows this stabilization strategy works across diverse architectures beyond convolutional networks.
Meng: If it scales well and handles different architectures, that makes it much more attractive for production environments where we have many different model types to maintain.
Lalam: The authors also flag some limitations; they mentioned that the framework can be sensitive to abrupt learning rate schedules, requiring a short adaptation period for the intrinsic parameters 'a' and 'b' to re-adjust.
Tom: And there’s another one too—the paper says that when you are jointly optimizing both the weight parameters and these activation parameters, you need sufficient sample diversity for geometric plasticity to converge effectively.
Jane: So, while the performance numbers are strong, the authors warn us that we still have to be careful about how we schedule learning rates and how diverse our training data is if we want this method to work perfectly.
Title and authors: Lu: They also pointed out that they need native CUDA kernel implementations for SALU and its gated variants if people want to actually use them in practice on hardware efficiently.
Meng: That’s something I'll keep in mind for implementation because the efficiency of the kernel matters a lot when you are running large models on real traffic.
Lalam: To wrap up, the paper "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers" shows that replacing rigid statistical normalization with SALU and its variants gives us a way to handle stabilization through intrinsic geometry rather than external statistics.
Tom: It sounds like they’ve successfully shown that we can get competitive accuracy on CIFAR-one hundred without any explicit normalization layers at all, which is a pretty strong result.
Jane: They’ve also proven that this approach isn't just a trick for one architecture; it has been shown to work in Vision Transformers too, suggesting it could be more broadly useful.
Lu: The idea of total plasticity means we are learning how the signal propagates at every level, which is a big conceptual shift away from partitioning learnability across separate components.
Meng: For me, the practical implication is that if we can design an activation function that handles stabilization intrinsically without needing constant external tuning based on batch statistics, it makes model maintenance much less fragile.
Lalam: So to summarize the findings of "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," they introduce SALU for intrinsic signal stabilization through geometry-driven dynamics, and then build SWALU and GALU on top of it for learnable gating.
Tom: It seems like the main thing here is moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat idea.
Jane: It’s a neat idea because it connects stabilization directly to how we define our nonlinearities in neural networks, instead of treating them as separate tools.
Lu: The depth dependent stratification they observed is important because it shows that the network learns to use different geometric regimes for early and deep layers, which is a kind of internal structural learning.
Meng: So if we want to deploy this, we’re not just looking at one fixed activation; we’re looking at a system where the activation itself evolves its geometry during training based on the data it sees.
Lalam: We need to keep an eye on those limitations they mentioned about learning rate schedules and sample diversity, because that tells us when to expect performance to drop if we don't tune things right.
The paper's summary: Tom: So, to recap, SaluNet is proposing we ditch those traditional normalization layers like Batch Normalization and replace them with this single learnable activation function called SALU, which uses its own shape and scale to handle signal stabilization geometrically instead of relying on batch statistics.
Jane: That’s a big idea because it means the network learns how to stabilize itself intrinsically, rather than having an external layer do all the heavy lifting based on what's happening in the current batch.
Lu: Exactly. The paper introduces this concept of total plasticity where connections, stabilization, and gating dynamics are all learned through this geometry-driven process. It’s a unified approach to training deep networks that replaces those rigid statistical constraints with something neuron-level.
Meng: From my side at the startup, the main thing I’m interested in is how this affects deployment cost and stability at scale. If we can remove those external statistics, does that mean fewer moving parts in our inference pipeline?
Tom: It does; it means we remove those running means and variances you have to track constantly during training and inference. And the results on CIFAR-one hundred are pretty solid, hitting eighty-three point two five percent accuracy without any normalization layers.
Jane: That’s a high number for a model that doesn't even have a standard normalization layer in it, so what makes that result stand out to you?
Tom: What stands out is the robustness at batch size one. The paper shows SaluNet keeps strong performance even when you scale the batch size down to one, which is where most other normalized architectures start failing.
Lu: That’s because they’re showing that this learned geometry gives it a way to stay stable without needing those statistical buffers that collapse at small batches. And they analyze the learned geometry itself, and they find a depth dependent stratification.
Jane: Depth dependent? What does that mean for the network structure?
Lu: It means early layers tend to have strong saturation and narrow linear regimes, while deeper layers prioritize wide linear regimes and relaxed gating to maximize how much information they can hold. It’s like the network naturally structuring itself based on how deep it is.
Meng: So that means the model learns different geometric behaviors for different parts of its structure without us having to manually design those layer behaviors in advance? That sounds like a lot of flexibility, but I gotta ask, what are the caveats here?
Lu: The paper does flag some limitations; they say the framework can be sensitive to abrupt learning rate schedules and requires a short adaptation period for those intrinsic parameters 'a' and 'b' to readjust. They also need sufficient sample diversity for that geometric plasticity to converge well.
Tom: So it’s not a magic fix that works perfectly every single time; you still have to tune the learning rate schedule carefully and make sure your training data is diverse enough for the geometry to settle into its right shape.
Jane: That makes sense. It’s not just plugging in an activation function and hoping for the best; it’s about controlling that geometry through training dynamics, but you still have to manage the environment around it.
Lu: Yeah, and they also point out that for real-world hardware use, we still need native CUDA kernel implementations for SALU and its gated variants if we want to get the performance right on a GPU.
Meng: Right. So the idea is powerful conceptually—stabilization through geometry—but as an engineer, I’m looking at those implementation hurdles and the tuning requirements.
Tom: It really does seem like they’ve successfully moved the burden of stabilization from external statistical mechanisms into the activation function itself, which is a neat way to think about model stability.
Jane: It connects stabilization directly to how we define our nonlinearities in neural networks, instead of treating them as separate tools you bolt on later.
Lu: And that unified principle extends beyond CNNs; they showed SaluNet-T works well for Vision Transformers, suggesting this idea applies across different architectures under the same total plasticity concept.
Meng: If it scales and works across those different architectures, that makes it much more attractive for production environments where we have a mix of model types to maintain.
Tom: So the main thing here is moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat way to think about model stability.
Jane: It’s a neat idea because it connects stabilization directly to how we define our nonlinearities in neural networks, instead of treating them as separate tools you bolt on later.
Lu: And that unified principle extends beyond CNNs; they showed SaluNet-T works well for Vision Transformers, suggesting this idea applies across different architectures under the same total plasticity concept.
Meng: If it scales and works across those different architectures, that makes it much more attractive for production environments where we have a mix of model types to maintain.
The paper's improvements: Tom: So, we’re talking about how these improvements take SaluNet from just being an interesting concept to something practically useful in training models.
Jane: What are they suggesting is that by making the stabilization mechanism learnable through geometry, you unlock a whole new kind of control over the signal propagation inside the network.
Lu: They are showing that this approach allows for geometric plasticity, where the network learns its own optimal behavior by adjusting those shape and scale parameters dynamically during training.
Meng: I’m interested in what they suggest about gradient flow, because we always worry about optimization stability when you get rid of those standard normalization layers.
Tom: They suggest using a specific form of SALU where the derivative has a certain polynomial decay, like SALU prime equals SALU divided by x cubed times one over 'a squared', which helps preserve usable gradients over a wider dynamic range.
Jane: That’s important because fixed activations often cause those gradients to vanish or explode too quickly, and this mechanism keeps them in check better.
Lu: And they also discuss how the framework enables organic self normalization, where the final embedding layer naturally drives its output distributions toward a symmetric, quasi-Gaussian profile without needing an explicit normalization layer there.
Meng: That’s a big win for deployment because it means less overhead in our model architecture and we don't have to worry about that final layer's statistics during inference.
Tom: And they also point out the deep stratification again, meaning early layers handle saturation while deep layers use wider linear regimes for representation, which is how you get both strong detail and broad feature capture simultaneously.
Jane: That means the network learns a hierarchical way to process information based on its depth, which is really fascinating from a science communication standpoint.
Lu: It shifts the whole paradigm; instead of having external layers impose constraints on the signal, we are learning how the neuron itself responds and stabilizes its own output through these geometry-driven dynamics.
Meng: So if this works as advertised—no normalization, better stability at batch size one—does that mean we can cut down on other complex stabilizing components in our pipelines?
Tom: It means yes; it suggests that shifting the stabilization burden to intrinsic neuron level dynamics offers four fundamental advantages, including geometric plasticity and organic self normalization.
Jane: That's the core message: you get better performance and stability by learning how the signal propagates at every level, instead of just imposing a fixed statistical correction on top.
Lu: The implication is that we can design architectures where the stabilization mechanism is fully integrated into the activation function itself, unifying stabilization and gating dynamics into a single component.
Meng: So to summarize what they’re pushing: learnable geometry handles stabilization and gating without needing external batch statistics or affine parameters, leading to better stability at batch size one and better performance across different model types like Transformers.
Conclusion: Tom: So we’re wrapping up this look at "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers." The main idea is that by using SALU and its gated variants, the network learns how to stabilize itself geometrically instead of relying on external batch statistics.
Jane: That’s right. It shows that we can replace rigid statistical normalization with an intrinsic, learnable geometric control mechanism for signal propagation.
Lu: The paper concludes that this framework achieves state of the art performance on CIFAR-one hundred and even competitive accuracy on ImageNet-1K without any explicit normalization layers at all <ref:2606.02927#pg1>.
Meng: That’s a solid number, but I still see the caveat about needing sufficient sample diversity for that geometric plasticity to converge effectively, which means we can't just throw it into production and expect perfect results every time.
Tom: Exactly. It’s not just a trick; there are still tuning requirements like learning rate schedules and data diversity that need careful management for this to perform well across the board.
Jane: So what does this mean for the world of deep learning? It suggests that we can fundamentally shift how we think about model stability, moving away from imposed statistical corrections toward intrinsic neuron-level dynamics.
Lu: The implication is a more unified principle of total plasticity where connections, stabilization, and gating are all jointly learned through this geometry-driven process.
Meng: From an engineering standpoint at the startup level, if we can remove those external statistical layers without sacrificing performance on real traffic data, that really simplifies the entire deployment pipeline.
Tom: It means we can get better stability at batch size one and across different model types like Vision Transformers by applying this single unified principle of total plasticity.
Jane: It’s about moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat way to think about model stability.
Lu: The depth dependent stratification they observed is important because it shows that the network learns to use different geometric regimes for early and deep layers, which is a kind of internal structural learning.
Meng: So if we want to deploy this, we’re not just looking at one fixed activation; we’re looking at a system where the activation itself evolves its geometry during training based on the data it sees.
Tom: It seems like they’ve successfully shown that replacing rigid statistical normalization with SALU and its variants gives us a way to handle stabilization through intrinsic geometry rather than external statistics.
Jane: They’ve also proven that this approach isn't just a trick for one architecture; it has been shown to work in Vision Transformers too, suggesting it could be more broadly useful across diverse architectures.
Lu: The authors conclude that shifting the stabilization burden to intrinsic neuron level dynamics offers four fundamental advantages, including geometric plasticity and organic self normalization.
Meng: So to wrap up what we’ve heard on "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," it’s a powerful framework that removes external statistical constraints by learning stabilization through geometry.
Tom: It really does seem like they’ve successfully shown that we can get competitive accuracy on CIFAR-one hundred without any explicit normalization layers at all, which is a pretty strong result.
Jane: They’ve also proven that this approach isn't just a trick for one architecture; it has been shown to work in Vision Transformers too, suggesting it could be more broadly useful across diverse architectures.
Lu: The idea of total plasticity means we are learning how the signal propagates at every level, which is a big conceptual shift away from partitioning learnability across separate components.
Meng: For me, the practical implication is that if we can design an activation function that handles stabilization intrinsically without needing constant external tuning based on batch statistics, it makes model maintenance much less fragile.
Tom: So to summarize the findings of "Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers," they introduce SALU for intrinsic signal stabilization through geometry-driven dynamics, and then build SWALU and GALU on top of it for learnable gating.
Jane: It seems like the main thing here is moving the complexity of normalization from an external statistical mechanism into the activation function itself, which is a neat idea.
Lu: And that unified principle extends beyond CNNs; they showed SaluNet-T works well for Vision Transformers, suggesting this stabilization strategy generalizes across diverse architectures beyond convolutional networks.
More episodes
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI