The limits of exactness: On the failure of automatic differentiation in physics-informed machine learning
cs.LG
Submitted: 2026-09-27
Updated: 2026-09-30
Comments: 18 pages, 2 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: Automatic differentiation (AD) lets neural networks compute derivatives of governing equations to machine precision, and this precision has made it the computational backbone of physics-informed
Terminology
Abstract
Automatic differentiation (AD) lets neural networks compute derivatives of governing equations to machine precision, and this precision has made it the computational backbone of physics-informed machine learning. Yet exactness in the mathematical sense is not the same as fidelity to the physics. Here I argue that a derivative can be numerically perfect and still be the wrong derivative for the problem at hand, because AD, by construction, has no notion of the physical structure a solution must obey. Convection and its associated directionality, diffusion, and dispersion are only the most visible instances of a much longer list that spans all branches of computational science and engineering, including conservation, thermodynamic consistency, symmetry, symplectic structure, positivity, monotonicity, and boundedness. Recognizing this broader gap reframes how the field should build the next generation of PDE-driven neural surrogates.
Sources
- GenVoid: Uncertainty-Aware Learning of Subsurface Material Defects with an Experimentally Validated Physics-Informed Generative Model
- Lagrangian Neural Networks
- Symplectic Recurrent Neural Networks
- Solution multiplicity and effects of data and eddy viscosity on Navier-Stokes solutions inferred by physics-informed neural networks
- Tackling Failure Modes of PINNs and PIKANs Using Conflict-Free Gradients
- An invariance constrained deep learning network for PDE discovery
- Improving physics-informed DeepONets with hard constraints
- Deep Neural Networks with Symplectic Preservation Properties
- Learning Hamiltonian Systems with Pseudo-symplectic Neural Network
- Symplectic Neural Operators for Learning Infinite Dimensional Hamiltonian Systems
- Neural non-canonical Hamiltonian dynamics for long-time simulations
- WANCO: Weak Adversarial Networks for Constrained Optimization problems
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks