Automated Numerical Stability Analysis of Deep Learning Operators
math.NA, cs.AI, cs.NA
Submitted: 2026-07-28
Updated: 2026-11-03
Code: https://github.com/chenxinye/noisefloat
License: http://creativecommons.org/licenses/by/4.0/
The gist: Finite-precision arithmetic unavoidably introduces numerical approximation errors.
Terminology
Abstract
Finite-precision arithmetic unavoidably introduces numerical approximation errors. Numerical computations may use insufficient precision or an improper formulation, which leads to numerical instability. In this paper, we introduce a unified software tool for stochastic numerical validation of deep-learning operators. The tool follows CESTAC on supported exposed operations and uses an operator-level data-perturbation approximation for GEMM-like kernels. Our developed software not only enables numerical validation with a single computation pass but also detects the sources of numerical instability and provides numerical stability monitoring during deep learning training and inference. We verified its effectiveness on the detection of polluted operators with injected numerical instabilities across various tasks. We believe that our developed method and tools provide valuable insights into developing numerically stable computing kernels, which are particularly critical for numerically stable and efficient deep learning training and inference.
Sources
- Mixed precision accumulation for neural network inference guided by componentwise forward error analysis
- Numerical stability analysis of large language models
- Mixed-Precision Lloyd's Algorithm for k-means Clustering
- Floating-point autotuning with customized precisions
- Verificarlo: checking floating point accuracy through Monte Carlo Arithmetic
- AdaQAT: Adaptive Bit-Width Quantization-Aware Training
- Deep Speech: Scaling up end-to-end speech recognition
- A Framework for Semi-Automatic Precision and Accuracy Analysis for Fast and Rigorous Deep Learning
- To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability
- Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
- Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Related papers
- Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm
- A Neural-preconditioned Poisson Solver for Mixed Dirichlet and Neumann Boundary Conditions
- Advanced Linear Algebra with Applications - Part I (Numerical linear algebra for PDEs, machine learning, and data assimilation)
- Operator learning meets inverse problems: A probabilistic perspective
- Second-order consistency for learning chaotic dynamics via randomized Jacobian matching
- Windowed thinning and query complexity for the bouncy particle and Zigzag samplers