A samplewise backpropagation method for neural networks driven by fractional Brownian motion

The gist In this paper, they develop a fractional stochastic neural network with residual dynamics driven by fractional Brownian motion and construct an adjoint recursion using a discrete stochastic maximum principle to prove mean square convergence for projected samplewise stochastic gradient descent

Model Formulation and Dynamics

The paper develops a fractional stochastic neural network with residual dynamics driven by fractional Brownian motion The forward propagation of the FSNNs is the controlled stochastic difference equation Xn+1 = Xn + h b(n, Xn, un) + √h σ(n, Xn, un) ξH n, where Xi0 = x0 and n = 0 to N-1. The diffusion coefficient sigma is permitted to depend jointly on (x, u), and the noise term is equivalently h 1/2-H σ(n, Xn, un) ΔB H n in raw fBm increments. The model is examined on regression, long memory time series and image classification problems.

Adjoint Equation and Optimality

The paper introduces a discrete stochastic maximum principle for the network This leads to the construction of an adjoint recursion, which is represented by the backward stochastic difference equation p n = An̄p nΘn(ζH n̄p n + β(n, n)q n) + rn, where M and Θ are bounded dN-measurable matrices and rn ∈ L2(Ω, n; Rd). The associated conditional noise projection is ξ n = ζH n̄p n + β(n, n)q n.

Samplewise Gradient Estimation

The paper derives a pathwise reverse sweep to obtain the samplewise gradient estimator, defined by ̂g n = b(nu, Xnu, un)∗̂p n+1 + λu(nu) + h-1/2 σu(nu, Xnu, un)∗̂p n+1 ξH n. This estimator is unbiased for deterministic parameter sequences u ∈ det, meaning E[̂g(u)] = ∇J(u) in det. The training algorithm uses one forward and one backward sweep per sampled trajectory.

Convergence Analysis

The paper proves a mean square convergence bound for projected samplewise SGD on deterministic network parameters Theorem 5.8 states that for the unique minimiser u∗, the error satisfies E‖‖u K - u∗‖2det ≤ 1 + VH(N) K(C0 + C1 h). Corollary 5.9 shows that under control independent diffusion, the variance term vanishes, leading to a bound of E‖‖u K - u∗‖2det ≤ C(1 + VH(N)) K.

Numerical Experiments

The numerical experiments are divided into three sets: E1 tests projected SGD convergence and uncertainty quantification, E2 isolates the long memory modelling effect by comparing FSNNs with a Brownian SDE-Net and an RNN on synthetic and real time series, and E3 studies classification robustness under common image corruptions and structured perturbations. The results identify settings in which fractional drivers improve long memory recovery or robustness relative to Brownian and deterministic baselines.

Conclusion

In this paper, they developed a numerical analysis framework for training fractional stochastic neural networks driven by normalised fractional Brownian increments They proved a mean square convergence bound for projected samplewise SGD on deterministic network parameters and reported numerical experiments on convergence, uncertainty quantification, long memory modelling and robustness. The results identify settings in which fractional drivers improve long memory recovery or robustness relative to Brownian and deterministic baselines.

How it works

1. The forward propagation of the FSNNs is the controlled stochastic difference equation Xn+1 = Xn + h b(n, Xn, un) + √h σ(n, Xn, un) ξH n.

2. The adjoint equation is derived as a backward stochastic difference equation p N = Φx(XN), p n = An̄p nΘn(ζH nH̄p n + β(n, n)q n) + rn, where M and Θ are bounded dN-measurable matrices and rn ∈ L2(Ω, n; Rd).

3. The samplewise gradient estimator is obtained by differentiating the realised single trajectory loss, defined by ̂g n = b(nu, Xnu, un)∗̂p n+1 + λu(nu) + h-1/2 σu(nu, Xnu, un)∗̂p n+1 ξH n.

4. The training procedure is a projected SGD driven by the samplewise gradient, updating u k+1 = Πdet(u k - η̂g k), where Πdet denotes the projection in the det norm of equation (16).

Key Findings

** The paper proves a mean square convergence bound for projected samplewise SGD on deterministic network parameters, showing that the error satisfies E‖‖u K - u∗‖2det ≤ 1 + VH(N) K(C0 + C1 h). The term VH(N) records the one step predictability of ξH n from its past and is bounded by one for every finite depth. 2.3. Underlying White Noise and the Matrices β, γ**

** The discrete fractional Brownian increments ξH n are decomposed into a predictable part and a fresh white noise innovation, enabling the derivation of an adapted backward stochastic difference equation This decomposition is crucial because the usual Brownian adjoint recursion based on independent increments cannot be transferred unchanged for fractional Brownian increments. 2.4. Backward Stochastic Difference Equations**

** The well-posedness of the fixed depth BSΔE is established by Lemma 2.3, which guarantees a unique adapted solution (p n,̄p n, q n) with finite moments of every order. 4.2. Discrete Stochastic Maximum Principle**

** The grid level stochastic maximum principle provides the optimality system where the Riesz gradient contains the factor h-1/2 σu(n)**

Experimental Validation

The experiments demonstrate that matching the driver Hurst exponent helps recover long memory scaling in synthetic fractional Ornstein–Uhlenbeck series with small errors The FSNNs show competitive or best performance on real data, particularly in long lag dependence on SPX, NhemiTemp, NBSdiff1kg and ethernet traffic. The results also show that the useful Hurst regime depends on both the data set and the perturbation spectrum.

Key Findings

** The paper proves a mean square convergence bound for projected samplewise SGD on deterministic network parameters, showing that the error satisfies E‖‖u K - u∗‖2det ≤ 1 + VH(N)

1. textbf Fractional Stochastic Neural Networks (FSNNs) for Long Memory Modeling: Improving Time Series Generation and Prediction Accuracy in E2/Table 4 metrics by matching the Hurst exponent to target long memory scaling with much more accuracy than both baselines. This allows AI systems to generate synthetic time series or predict real-world data with persistent increments, as evidenced by the significant reduction in Hurst error and improvement in ACF compared to Brownian SDE-Nets or RNNs when using an estimated target Hurst exponent (e.g., 0.7 for SPX log close).

2. textbf Robustness to Structured Additive Noise: Enhancing Image Classification Under Perturbations (E3/Table 6) by leveraging stochastic backbones that show superiority over deterministic CNNs and SNNs against two-dimensional fBm random fields at amplitude 0.2, indicating a more resilient architecture for structured noise environments. This enables AI models to maintain higher accuracy under controlled additive noise fields than standard architectures, especially in Fashion-MNIST and CIFAR-10 tasks.

3. textbf Samplewise Gradient Unbiasedness: Enabling Reliable Training via Pathwise Gradient Estimation (Theorem 4.3/Corollary 5.5) by utilizing the pathwise reverse sweep to compute the gradient, which is shown to be unbiased: D J(u)v = E [hN∑−1 n=0⟨̂gn, vn⟩] for deterministic parameters. This provides a theoretically sound method for training network weights using a single trajectory of noise realizations, reducing reliance on computationally expensive full adjoint solutions.

4. textbf Projected SGD Convergence Guarantees: Ensuring Stable Optimization in High-Dimensional Parameter Spaces (Theorem 5.8) by proving the convergence bound: E‖‖uK − u∗‖‖2det≤1 + VH(N)K(C0 + C1h). This allows for the deployment of projected stochastic gradient descent on deterministic parameters, providing a rigorous guarantee on how many iterations are needed to reach an error bound proportional to the architecture-dependent term (e.g., 1 + VH(N)K).

5. textbf Control-Independent Diffusion Scaling: Improving Efficiency in Training via Variance Reduction (Corollary 5.9) by showing that when the diffusion is control independent, the h−1 variance term vanishes and E‖‖uK − u∗‖‖2det≤C(1 + VH(N))K. This results in a sharper iteration bound, where the optimization error scales linearly with the iteration number (K), simplifying complexity for control-independent diffusion models.