Linear equivalence of nonlinear recurrent neural networks

arXiv:2604.23489 · cond-mat.dis-nn, q-bio.NC · Submitted 2026-04-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.

Kai: We're looking at the paper: "Linear equivalence of nonlinear recurrent neural networks".

Mira: Here's the gist of the paper. Large nonlinear recurrent neural networks with random couplings generate rich, potentially chaotic activity and are of interest in neuroscience and other fields The covariance matrix is the key object encoding the structure of activity The paper derives an ansatz showing that at large N and for typical quenched couplings, this covariance matrix matches that of a linear network with the same couplings, driven by independent noise Model and Dynamics The study focuses on nonlinear recurrent networks of N units with activity variables ϕi(t), external drives ξi(t), and couplings Jij The single-unit dynamics are defined by a causal functional T

h: (t) where phi i(t) = T

hi: (t) and hi(t) = η i(t) + ξ i(t) The focus is on the rate model of Sompolinsky et al.

14: where T

h: (t) is defined implicitly by phi(t) = f(x(t)), ∂tx(t) = −x(t) + h(t), and f(x) is an odd-symmetric sigmoidal nonlinearity The couplings J play the role of quenched disorder, assumed to be i.i.d.

Mira: with first and second moments ⟨Jij ⟩J = 0, J 2 ij J = g squared N Covariance and Response Matrices The time-lagged covariance matrix has elements C phi ij (τ) = ⟨ϕi(t + τ) ϕj (t)⟩t The instantaneous functional derivatives S phi ij (t, t′) = δphi i(t)/δxi j (t′), and their time averages comprise the N × N response matrix S phi ij (τ) The single-site DMFT provides scalar order parameters around which the diagonal elements of the covariance matrix concentrate at large N The limiting values C phi⋆ (τ) and S phi⋆ (τ) are determined via a cavity method The Linear-Equivalent Ansatz The ansatz states that the covariance matrix of the nonlinear network takes the same form as Eq.

Kai: (15), with T (ω) replaced by S phi⋆ (ω) and T (ω) squared σ squared ξ(ω) replaced by C∆⋆(ω), defined below This is denoted by C¯ϕ(ω): Linear-equivalent ansatz (Shen and Hu, 2025) The diagonal elements are of order one and are approximated by C¯ϕ(ω) to O(N −1/2) accuracy, while the off-diagonal elements scale as N −1/2 and are approximated to O(N −1) accuracy in the root-meansquare (RMS) sense Derivation of Linear Equivalence The derivation shows that residual cross-covariances at distinct sites are suppressed below the scale of the activity cross covariances, so that the residuals act as independent noise driving a linear network The cavity method separates pairwise activity into Gaussian and non-Gaussian parts This leads to Property 1: joint Gaussianity of the cavity fields (Property 1) and decoupling of the kernel from cavity-field quantities (Property 2)

29: Higher-Order Moments Due to weak interactions, the moment-cumulant formula reduces higher-order cross-site moments at leading order to products of pairwise covariances controlled by the ansatz The linear-equivalent ansatz, together with single-site DMFT, also suffices for computing repeated-index even moments Numerical Verification and Conclusion The analytical results are verified through simulations of the model of Sompolinsky et al.

Title and authors: Kai: Moving on from the setup, "Linear equivalence of nonlinear recurrent neural networks" explores what this mathematical match actually means for these kinds of networks. It’s about showing that a complex nonlinear system can be described by a simpler linear one plus some independent noise.

Mira: What it means is that the complexity isn't just inherent to the nonlinearity itself, but rather how the disorder couples with it, and this paper shows that under certain conditions, that coupling simplifies dramatically.

Lev: For us in hardware terms, if we can reduce a nonlinear network to something linear driven by noise, it suggests we might be able to analyze the dynamics using established methods for linear systems before worrying about the full nonlinearity.

Kai: Right, so they're suggesting a way to understand the rich activity without having to simulate every single coupling configuration individually.

Mira: And they do this by showing that residual cross covariances between different sites are strongly suppressed, essentially acting like independent noise driving that simpler linear network.

Lev: That suppression is important because it means we only need to focus on these leading order terms when trying to understand the activity structure at a large number of units.

The paper's summary: Kai: So, this paper summarizes how they use that two site cavity method to show this linear equivalence. They decompose each unit's activity into two parts: a linear response and a nonlinear residual.

Mira: And then using the cavity method, they demonstrate that the cross covariances of those residuals at different sites are suppressed below the scale of the main activity cross covariances.

Lev: That suppression is crucial because it means these residual terms effectively behave like independent noise driving a linear network, which simplifies things significantly for analysis.

Kai: They also introduce this linear-equivalent ansatz, which is denoted by C bar phi omega, where they replace certain parts of the original nonlinear system with these cavity quantities.

Mira: The result is that the diagonal elements of this covariance matrix are of order one, and they approximate those using this ansatz to a precision of O N minus one over N squared accuracy <ref:2604.23489#pg1>.

Lev: And for the off-diagonal elements, they get an RMS accuracy of O N minus one relative to their scale, which is significant because that means the cross terms are well-controlled <ref:2604.23489#pg1>.

The paper's improvements: Kai: One of the major improvements they highlight is how this ansatz lets them compute higher-order moments more efficiently. They show that due to weak interactions, the moment-cumulant formula reduces these higher-order cross site moments to just products of pairwise covariances controlled by that linear equivalence ansatz.

Mira: That means you don't need to track all those complicated four point or six point correlations; you can get them by just knowing the two point covariance information from the ansatz.

Lev: For running this on real hardware, it means we can estimate these higher-order statistics with less overhead because we only need to track the two site covariances instead of everything else.

Kai: And they also confirmed that this framework works for even moments, which is pretty neat because those are often the hardest ones to calculate in statistical physics problems.

Mira: They proved this using the linear-equivalent ansatz together with single site dmft, which helps them compute these repeated index even moments accurately.

Conclusion: Kai: So, to wrap up, this paper on "Linear equivalence of nonlinear recurrent neural networks" shows that we can approximate complex nonlinear dynamics by a simpler linear network driven by noise at large network sizes. This is achieved through the linear-equivalent ansatz.

Mira: The key finding is that the residuals between sites act like independent noise, which suppresses cross covariances, leading to a covariance matrix structure that's very close to its linear counterpart.

Lev: From an engineering standpoint, this gives us a tool to analyze the structure of activity in these large recurrent systems using techniques developed for simpler linear systems.

Kai: It suggests that the rich activity we see in these nonlinear networks can be understood by looking at their fluctuations as noise driving a linear system, which is a new way to look at things.

Mira: And they also rigorously established properties about the cavity fields themselves, showing joint Gaussianity and decoupling from the kernel.

Lev: So, for those of us building error correction codes for these systems, this provides a solid mathematical foundation to start analyzing the noise driving terms in these large networks.

Kai: That's where we'll leave it for now. Next up, we look at some work on three body corrections to Brownian viscosity.

Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University

cond-mat.dis-nn, q-bio.NC

Submitted: 2026-04-26

Updated: 2026-11-09

Comments: 49 pages, 10 figures, 2 tables; updated with extension to nonzero mean activities, three new models, corrected higher-order moments, and AI methodology statement

Code: https://github.com/davidclark1/Linear-Equivalence

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Large nonlinear recurrent neural networks with random couplings generate rich, potentially chaotic activity and are of interest in neuroscience and other fields The covariance matrix is the key…

Key concepts

Linear-Equivalent Ansatz (Cbar phi)
This is a mathematical model suggesting that the complex covariance matrix of a nonlinear network can be approximated by the covariance matrix of a simpler, linear network driven by independent noise. This approximation holds for large networks and helps simplify calculations in strongly nonlinear systems.
Cavity Method
A technique used to analyze the statistical properties of large neural networks by breaking them down into smaller, manageable pieces (cavities). It allows researchers to rigorously prove that certain joint Gaussianity and decoupling properties hold at leading order.
Response Matrix (S_phi)
This matrix describes how a change in one input affects all other units in the network. The paper shows that this response matrix can also be approximated by a linear-equivalent form, ensuring accurate characterization of both diagonal and off-diagonal elements.

Terminology

Summary

Large nonlinear recurrent neural networks with random couplings generate rich, potentially chaotic activity and are of interest in neuroscience and other fields The covariance matrix is the key object encoding the structure of activity The paper derives an ansatz showing that at large N and for typical quenched couplings, this covariance matrix matches that of a linear network with the same couplings, driven by independent noise

Model and Dynamics

The study focuses on nonlinear recurrent networks of N units with activity variables ϕi(t), external drives ξi(t), and couplings Jij The single-unit dynamics are defined by a causal functional T h where phi i(t) = T hi and hi(t) = η i(t) + ξ i(t) The focus is on the rate model of Sompolinsky et al. [14] where T h is defined implicitly by phi(t) = f(x(t)), ∂tx(t) = −x(t) + h(t), and f(x) is an odd-symmetric sigmoidal nonlinearity The couplings J play the role of quenched disorder, assumed to be i.i.d. with first and second moments ⟨Jij ⟩J = 0, J 2 ij J = g squared N

Covariance and Response Matrices

The time-lagged covariance matrix has elements C phi ij (τ) = ⟨ϕi(t + τ) ϕj (t)⟩t The instantaneous functional derivatives S phi ij (t, t′) = δphi i(t)/δxi j (t′), and their time averages comprise the N × N response matrix S phi ij (τ) The single-site DMFT provides scalar order parameters around which the diagonal elements of the covariance matrix concentrate at large N The limiting values C phi⋆ (τ) and S phi⋆ (τ) are determined via a cavity method

The Linear-Equivalent Ansatz

The ansatz states that the covariance matrix of the nonlinear network takes the same form as Eq. (15), with T (ω) replaced by S phi⋆ (ω) and T (ω) squared σ squared ξ(ω) replaced by C∆⋆(ω), defined below This is denoted by C¯ϕ(ω): Linear-equivalent ansatz (Shen and Hu, 2025) The diagonal elements are of order one and are approximated by C¯ϕ(ω) to O(N −1/2) accuracy, while the off-diagonal elements scale as N −1/2 and are approximated to O(N −1) accuracy in the root-meansquare (RMS) sense

Derivation of Linear Equivalence

The derivation shows that residual cross-covariances at distinct sites are suppressed below the scale of the activity cross covariances, so that the residuals act as independent noise driving a linear network The cavity method separates pairwise activity into Gaussian and non-Gaussian parts This leads to Property 1: joint Gaussianity of the cavity fields (Property 1) and decoupling of the kernel from cavity-field quantities (Property 2) [29]

Higher-Order Moments

Due to weak interactions, the moment-cumulant formula reduces higher-order cross-site moments at leading order to products of pairwise covariances controlled by the ansatz The linear-equivalent ansatz, together with single-site DMFT, also suffices for computing repeated-index even moments

Numerical Verification and Conclusion

The analytical results are verified through simulations of the model of Sompolinsky et al. [14] with errorfunction nonlinearity f(x) = erf(√πx/2), Gaussian i. p. 6i.d. couplings satisfying Eq. p. 6 (4), and no external drive (ξi(t) = 0) The linear-equivalent ansatz reproduces off-diagonal elements to O(N −1) RMS accuracy, or O(N −1/2) relative to the off-diagonals’ N −1/2 scale The present work derives the conjecture

Appendix A: Cumulant Scalings and Cavity Properties

The cumulant scalings underlying the two-site cavity method show that each additional distinct site suppresses the cumulant by N −1/2 Property 1 states that joint Gaussianity of the cavity fields (Property 1) Property 2 states that the F-kernel decouples from cavity-field functionals at leading order

Appendix D: Response Matrix

The response matrix S phi(ω) admits an analogous leading-order approximation S phi⋆ (ω)M(ω) with the same element-wise precision as C¯ϕ(ω) The off-diagonal RMS is O(N −1), and the diagonal precision follows from concentration of C x ii(ω) around C x⋆ (ω)

Appendix E: Alternative Derivation

The self-consistent equation derived from the cavity method, Eq. (E17), has the same structure as the linear-network self-consistent equation (45) at leading order The solution is the linear-network covariance matrix (15) under those replacements, the ansatz C¯ϕ(ω) = C∆⋆(ω)M(ω)M(ω)†

Appendix F: Sompolinsky-model Specialization and Preactivation Covariance

For the rate model of Eq. (3), the covariance ansatz becomes C¯phi(ω) = C phi⋆ (ω)(1 − g squared β 2⋆ 1 + ω 2) I − β⋆ 1 + iω J−1 I − β⋆ 1 + iω J−† The diagonal precision of O(N −1/2) follows from concentration of C x ii(ω) around C x⋆ (ω)

Appendix H: Numerical Details

The simulations use the error-function nonlinearity f(x) = erf(√πx/2), which matches tanh(x) in slope at the origin (f'(0) = 1), and likewise saturates at f(±∞) = ±1, while permitting closed-form evaluation of the Gaussian averages arising in the single-site DMFT The theory prediction C¯phi(ω) (Eq. (16)) is obtained by first solving the single-site DMFT self-consistently for C phi⋆ (τ) using the “particle in a potential” method of Sompolinsky et al. p. 1 [14] The inverse Fourier transform to C¯phi(τ) at the simulation output lags, extending up to τmax = 100, is performed by a direct sum over frequency bins

Discussion

The nonlinear network achieves this without fine-tuning [17] and can perform pattern separation through its recurrent dynamics The linear equivalence offers an alternative, in which the same signature arises from a weakly chaotic nonlinear network generating its own fluctuations, without fine-tuning The nonlinear network has a positive Lyapunov exponent (in fact, an extensive number of them [35]), and can therefore perform pattern separation through its recurrent dynamics

Relation to Concurrent Work

Our approach is organized around cavity quantities, each of which is a random projection of reservoir trajectories that are independent of the projecting couplings The cavity-variable properties constitute natural assumptions, since they are what one obtains by naively not distinguishing t-averaging from J-averaging and applying the central limit theorem

Relation to Linear and Nonlinear Views

Near the chaotic transition, the nonlinear and linear-equivalent pictures are dual views of the same critical point The nonlinear network has a positive Lyapunov exponent (in fact, an extensive number of them [35]), and can therefore perform pattern separation through its recurrent dynamics, which a stable linear system cannot The nonlinear network achieves this without fine-tuning [17], since geff stays close to 1 throughout the chaotic regime (e.g., geff → 1/√π − 2 ≈ 0. p. 794 as g → ∞ for f(x) = tanh(x)) [15]

The approach uses a path integral to expand a given joint moment as a power series in J around the DMFT background The counting argument analogous to that in Appendix A shows that the Wick decomposition holds at leading order, and the two-point factors are then explicitly resummed Our approach is organized around cavity quantities, each of which is a random projection of reservoir trajectories that are independent of the projecting couplings

Related Equivalence Results

The result is analogous to a growing body of linear equivalence results in machine learning We derive an equivalent linear-plus-noise system, starting from a nonlinearity acting on O(1) elements In the recurrent setting, the disorder and the activity it generates are coupled, an aspect that we handled using the cavity method

Acknowledgments

It is a pleasure to thank Ashok Litwin-Kumar, L.F. Abbott, Jacob Zavatone-Veth, Sebastian Mizera, Yu Hu, Xuanyu Shen, Yue M. p. 8 Lu, Alexander van Meegen, Blake Bordelon, and Cengiz Pehlevan for helpful discussions The author is especially grateful to Albert Wakhloo whose diagrammatic treatment preceded the calculations presented here The author is supported by a Kempner Institute Research Fellowship

References

[1] A. Arieli, A. Sterkin, A. p. 8 Grinvald, and A. p. 8 Aertsen, Science 273, 1868 (1996)

[2] H. Jaeger and H. p. 8 Haas, science 304, 78 (2004)

[3] S. Takasu and T. p. 8 Aoyagi, Physical Review Research 7 (2025)

[4] D. G. Clark, B. Bordelon, J. p. 8 A. Zavatone-Veth, and C. p. 8 Pehlevan, bioRxiv, 2026 (2026)

[5] E. M. Trautmann, J. p. 8 K. Hesse, G. M. Stine, R Xia, S Zhu, D J O’Shea B Karsh J Colonell F F Lanfranchi S Vyas et al. p. 8, Nature Neuroscience 28, 1562 (2025)

[6] J Manley, S Lu K Barber J Demas H Kim D Meyer F M Traub and A Vaziri Neuron 112, 1694 (2024)

[7] S Chung and L F Abbott Current opinion in neurobiology 70, 137 (2021)

[8] J P Cunningham and B M Yu Nature neuroscience 17, 1500 (2014)

[9] P Gao and S Ganguli Current opinion in neurobiology 32, 148 (2015)

[10] C Stringer M Pachitariu N Steinmetz M Carandini and K D Harris Nature 571, 361 (2019)

[11] D A Pospisil and J W Pillow Proceedings of the National Academy of Sciences 122, e250653535122 (2025)

[14] H Sompolinsky A Crisanti and H-J Sommers Physical review letters 61, 259 (1988)

[15] D G Clark L Abbott and A Litwin-Kumar Physical Review Letters 131, 118401 (2023)

[16] D G Clark O Marschall A Van Meegen and A LitwinKumar Physical Review X 15, 041019 (2025)

[17] X Shen and Y Hu arXiv preprint arXiv:2508.05288 (2025)

[18] Code for this paper is available at https://github.com/davidclark1/Linear-Equivalence (2026)

[19] H Sompolinsky and A Zippelius Physical Review Letters 47, 359 (1981)

[20] L F Cugliandolo and J Kurchan Physical Review Letters 71, 173 (1993)

[21] F Mignacco F Krzakala P Urbani and L Zdeborová Advances in Neural Information Processing Systems 33, 9540 (2020)

[26] P Gao E Trautmann B Yu G Santhanam S Ryu K Shenoy and S Ganguli BioRxiv, 214262 (2017)

[37] J Pennington and P Worah Advances in neural information processing systems 30 (2017)

[38] C Louart Z Liao R Couillet The Annals of Applied Probability 28, 1190 (2018)

[39] S Goldt B Loureiro G Reeves F Krzakala M Mézard and L Zdeborová in Mathematical and Scientific Machine Learning (PMLR, 2022) pp. 426–471 (2022)

[40] H Hu Y M Lu IEEE Transactions on Information Theory 69, 1932 (2022)

[41] D J Thouless P W Anderson and R G Palmer Philosophical Magazine 35, 593 (1977)

[42] J Aljadeff M Stern and T Sharpee Physical review letters 114, 088101 (2015)

[43] D G Clark and M Beiran Proceedings of the National Academy of Sciences 122, e2404039122 (2025)

[45] N Alexeev F Götze and A Tikhomirov Lithuanian mathematical journal 50, 121 (2010)

[46] J A Zavatone-Veth and C Pehlevan SciPost Physics Core 6, 026 (2023)

[47] M Stern H Sompolinsky and L F Abbott Physical Review E 90, 062710 (2014)

[48] D G Clark L Abbott and A Litwin-Kumar Physical Review X 15, 041019 (2025)

[49] G Bunin Physical Review E 95, 042414 (2017)

[50] F Roy G Biroli G Bunin and C Cammarota Journal of Physics A: Mathematical and Theoretical 52, 484001 (2019)

Here we derive the cumulant scalings underlying the two-site cavity method We first derive the scaling of cumulants of reservoir variables, showing that each additional distinct site suppresses the cumulant by N −1/2 The cavity scalings can in fact be obtained from the reservoir cumulant scalings alone, since the cavity suppression is in essence a central limit theorem for quenched random projections of weakly correlated variables

Appendix B: Numerical Verification of Cumulant Scalings and Cavity Properties

We test the cumulant scalings of Eqs. (A16) and (A12), together with the resulting Properties 1 and 2, in large-N simulations of the dynamics of Eqs. (1) and (2) at g = 20, where non-Gaussian effects on network statistics are most pronounced [15] The empirical cavity cumulants follow the predicted rates

Appendix C: Cumulant Expansions

For jointly Gaussian (X, Y), cumulants of order ≥ 3 vanish and Eq. (C2) collapses to its (p, q) = (1, 1) term, giving Price’s theorem For such Gaussian fields η1(t), η2(t), in Fourier space we have Price (functional): ⟨Fη1 Gη2∗⟩ η1,η2 = δF[η1]δh1/η1(ω)δG[eta2]d h2/eta2(ω)∗ C eta 12(ω) + O((C eta 12) 2), where the second term is the Furutsu–Novikov theorem

The response matrix S phi(ω) admits an analogous leading-order approximation S¯phi(ω) with the same element-wise precision as C¯phi(ω) The off-diagonal RMS is O(N −1), and the diagonal precision follows from concentration of C x ii(ω) around C x⋆ (ω)

The cavity pair can be any pair of units, this extends to the matrix equation C phi(ω) = S phi⋆ (ω) squared J C phi(ω) J T + C∆⋆(ω) S phi(ω)F(ω)√N + S phi(ω)∗ F(ω) T√N + I + E C(ω), where E C(omega) has diagonal elements of O(N −1/2) and off-diagonal elements of O(N −1), and the identity term ensures the correct diagonal

The covariance ansatz Eq. (16) becomes C¯phi(ω) = C phi⋆ (ω)(1 − g squared β 2⋆ 1 + ω 2) I − β⋆ 1 + iω J−1 I − β⋆ 1 + iω J−† The diagonal precision of O(N −1/2) follows from concentration of C x ii(ω) around C x⋆ (ω)

Appendix G: Other Models within the Same Framework

The framework of Eqs.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Matrix Covariance Approximation for Nonlinear Networks: The system can accurately approximate nonlinear recurrent neural networks by using a linear network driven by independent noise, as shown in Eq. (15), which is the linear-equivalent ansatz Cbar phi (omega). This allows for the application of random matrix theory to strongly nonlinear systems, providing analytical tractability where exact solutions are unavailable.

  2. Off-Diagonal Cross Covariance Prediction: The improved system can predict off-diagonal elements of the covariance matrix with an RMS accuracy of O(N - 1) relative to the N - 1/2 scale, as derived in Eq. (19). This specific precision is achieved by accounting for nonGaussian contributions via the two-site cavity method and showing that residual cross covariances are strongly suppressed.

  3. Effective Drive Generation: The AI system can model how nonlinearity generates an effective external drive in the linear-equivalent network, which is the same conclusion as the residual-as-noise picture of the first derivation. This allows for a richer understanding of intrinsic dynamics without relying on an external drive, as suggested by the observation that the nonlinear network has a positive Lyapunov exponent (in fact, an extensive number of them [35]).

  4. Higher-Order Moment Reduction: The system can compute higher-order cross-site moments by decomposing them into products of pairwise covariances at leading order, reducing them to the linear-equivalent ansatz Cbar phi (omega). This allows for the accurate calculation of complex statistics like four-point correlations using only two-point covariance information.

  5. Cavity Field Properties Analysis: The system can rigorously analyze cavity field properties, demonstrating that joint Gaussianity of the cavity fields and decoupling of the kernel from cavity-field quantities hold at leading order, as established by Properties 1 and 2 derived in Appendix A. This provides a robust mathematical framework for analyzing joint statistics at pairs of sites.

  6. Response Matrix Approximation: The system can approximate the nonlinear response matrix S phi (omega) using an ansatz Sbar phi (omega) = S phi star (omega) M(omega), which possesses the same element-wise precision as the covariance-matrix ansatz Cbar phi (omega). This ensures that both diagonal and off-diagonal elements of the response matrix are accurately characterized.

Sources

Related papers