Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations
cs.NE, cs.LG
Submitted: 2024-11-22
Updated: 2025-06-13
Journal ref: Computer Physics Communications 315 (2025) 109753
DOI: 10.1016/j.cpc.2025.109753
License: http://creativecommons.org/licenses/by/4.0/
The gist: Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs).
Terminology
Abstract
Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs). However, they face challenges related to spectral bias (the tendency to learn low-frequency components while struggling with high-frequency features) and unstable convergence dynamics (mainly stemming from the multi-objective nature of the PINN loss function). These limitations impact their accuracy for problems involving rapid oscillations, sharp gradients, and complex boundary behaviors. We systematically investigate learnable activation functions as a solution to these challenges, comparing Multilayer Perceptrons (MLPs) using fixed and learnable activation functions against Kolmogorov-Arnold Networks (KANs) that employ learnable basis functions. Our evaluation spans diverse PDE types, including linear and non-linear wave problems, mixed-physics systems, and fluid dynamics. Using empirical Neural Tangent Kernel (NTK) analysis and Hessian eigenvalue decomposition, we assess spectral bias and convergence stability of the models. Our results reveal a trade-off between expressivity and training convergence stability. While learnable activation functions work well in simpler architectures, they encounter scalability issues in complex networks due to the higher functional dimensionality. Counterintuitively, we find that low spectral bias alone does not guarantee better accuracy, as functions with broader NTK eigenvalue spectra may exhibit convergence instability. We demonstrate that activation function selection remains inherently problem-specific, with different bases showing distinct advantages for particular PDE characteristics. We believe these insights will help in the design of more robust neural PDE solvers.
Sources
- KAN: Kolmogorov-Arnold Networks
- Challenges in Training PINNs: A Loss Landscape Perspective
- On the expressiveness and spectral bias of KANs
- On the Activation Function Dependence of the Spectral Bias of Neural Networks
- Searching for Activation Functions
- Learning Specialized Activation Functions for Physics-informed Neural Networks
- KAN versus MLP on Irregular or Noisy Functions
- Adaptive Training of Grid-Dependent Physics-Informed Kolmogorov-Arnold Networks
- Kolmogorov Arnold Informed neural network: A physics-informed deep learning framework for solving forward and inverse problems based on Kolmogorov Arnold Networks
- Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximation
- Kolmogorov-Arnold Networks are Radial Basis Function Networks
- BSRBF-KAN: A combination of B-splines and Radial Basis Functions in Kolmogorov-Arnold Networks
- Enhancing Graph Collaborative Filtering with FourierKAN Feature Transformation
- Wav-KAN: Wavelet Kolmogorov-Arnold Networks
- Unveiling the Power of Wavelets: A Wavelet-based Kolmogorov-Arnold Network for Hyperspectral Image Classification
- SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions
- Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
Related papers
- Evolutionary Ensemble of Agents
- EvoDR: Evolving Dispatching Rules via Large Language Model for Dynamic Flexible Assembly Flow Shop Scheduling
- RECAP: Local Hebbian Prototype Learning as a Self-Organizing Readout for Reservoir Dynamics
- Encoding and Decoding Temporal Signals with Spiking Bandpass Wavelets
- Design and Analysis of Temporal Representations in Neural Networks
- Evolutionary Computation: A Paradigm of Natural Generative AI and Creative Synthesis