Embedding stochastic dynamics in latent manifolds of recurrent neural networks

arXiv:2602.14885 · cond-mat.dis-nn, cond-mat.stat-mech, cs.LG, q-bio.NC · Submitted 2026-02-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.

Kai: We're looking at the paper: "Embedding stochastic dynamics in latent manifolds of recurrent neural networks".

Mira: Here's the gist of the paper. Recurrent neural networks can faithfully embed the drift and diffusion of given stochastic differential equations, including nonlinear and nonequilibrium dynamics such as chaotic attractors, by training them to represent arbitrary nonlinear SDEs within a low-dimensional latent subspace <ref:2602.14885#pg2>.

Kai: This framework extends attractor neural network theory beyond equilibrium by allowing asymmetric connectivity to implement a broad class of dynamical computations within low-dimensional manifolds <ref:2602.14885#pg4>.

Mira: Drift-Diffusion Matching Framework The core methodology introduced is the drift–diffusion matching (DDM) framework, which trains continuous-time RNNs to represent arbitrary nonlinear SDEs with given drift and diffusion coefficients within a low-dimensional latent subspace <ref:2602.14885#pg4>.

Kai: The training involves directly matching the drift and diffusion of the target process to the RNN's dynamics in an affine subspace <ref:2602.14885#pg4>.

Title and authors: Kai: So, we're diving into the paper "Embedding stochastic dynamics in latent manifolds of recurrent neural networks." It’s about using recurrent neural networks to model any nonlinear stochastic differential equation, given its specific drift and diffusion coefficients.

Mira: Basically, they take those complicated equations—like things with chaotic attractors—and they try to fit them inside a low-dimensional space that the network can handle.

Lev: It sounds like they’re trying to distill a massive amount of continuous system behavior down into something manageable for a neural network.

Kai: Exactly. The core idea is the drift–diffusion matching framework, which trains these recurrent neural networks to represent those arbitrary nonlinear stochastic differential equations within that low-dimensional subspace.

Mira: The training process involves directly matching the drift and diffusion terms of the target system to what the recurrent neural network does in a specific affine subspace.

Lev: So instead of just tweaking a few parameters one by one, they are training the entire network structure to reproduce those complex movements described by those drift and diffusion coefficients all at once. That sounds like a lot of simultaneous constraints on the system.

Kai: Right. They reformulate this training as approximating a target vector field using a two-layer perceptron structure, which is how they achieve that alignment in practice, like what you'd do to fit a function.

Mira: Once trained, you can take that network and integrate it over time to get the actual stochastic dynamics of your target system when you project it into the learned subspace.

Lev: That integration part is what makes it powerful; if the network learns the right mapping, you can simulate trajectories from a real stochastic system just by looking at the projected dynamics. But I'm still worried about computational cost. If we have to integrate over time for every single trajectory we want to check, how fast is that actually going to run on real hardware?

Kai: They look at this process on networks with, say, fifty or sixty four neurons and they show traces from those networks trained to approximate systems like the van der Pol oscillator. They show that when you project the dynamics into the learned subspace, you recover a trajectory from the stochastic van der Pol system that was embedded in that network architecture.

Mira: That proves it's feasible for smaller systems, showing that even with a limited number of neurons, you can capture these specific nonlinear behaviors under these conditions. So this paper is setting up the framework for how we model dynamics in latent spaces.

The paper's summary: Lev: So the paper doesn't just stop at fitting the basic dynamics; they suggest ways to make this framework handle more complex behaviors. What are these specific enhancements they propose for moving beyond simple models?

Kai: They introduce two distinct classes of dynamics that go beyond just simple gradient flows or purely time-reversible systems, which is a significant step because most classical models stick to those simpler types.

Mira: The first one is input-driven attractor switching. This uses network inputs to switch between the different minima of an energy potential, which allows the system to retrieve a specific memory based on what's coming in from sensory information or other neural circuits.

Lev: Switching attractors sounds like it requires some kind of active control mechanism built into the system, not just something that passively follows a simple downhill path.

Kai: They suggest this switching is achieved by using network inputs to select which minimum gets prioritized, essentially tilting the potential so that one of those minima becomes the global minimum.

Mira: Then they have attractor-cycling stationary diffusions. These use irreversible rotational currents to cycle between minima in a preferred direction, which models a memory system autonomously transitioning between related memories.

Lev: Rotational currents sound interesting for modeling how a memory system could transition between states on its own, instead of just settling into one stable minimum.

Kai: And for those cycling dynamics, they construct a rotational field that pushes trajectories around a differentiable curve through the minima, even when those minima are non-coplanar in three dimensions. This is done by defining a rotational vector field using an antisymmetric matrix.

Mira: That construction of the rotational field is interesting because it allows the system to cycle between attractors in a specified order using irreversible currents. It models autonomous transitions between related memories.

Lev: But that sounds computationally intensive for real hardware, especially when you have to manage those non-coplanar movements. The paper doesn't really address the overhead of that specific geometric construction.

Kai: They do point out that this means encoding rotational or cycling dynamics requires more network complexity, which shows up in larger weights in those asymmetric or irreversible parts of the network.

Mira: And they suggest a way to analyze this by splitting the network drift into reversible and irreversible components, allowing you to compare it with mathematical tools like the Helmholtz–Hodge decomposition, which gives you a trainable parameter between zero and one.

The paper's improvements: Kai: So to wrap up on "Embedding stochastic dynamics in latent manifolds of recurrent neural networks," the main point is how this framework allows recurrent neural networks to encode a whole range of nonlinear dynamics with specified drift and diffusion coefficients within those low-dimensional spaces.

Mira: They summarize that the drift–diffusion matching framework provides a straightforward way to encode low-dimensional computational dynamics with specific drift and diffusion coefficients in high-dimensional neural networks, connecting the math directly to the network structure.

Lev: From an engineering standpoint, if we can figure out how to efficiently reconstruct those affine subspaces using things like principal component analysis, we might be able to quantify the complexity of chaotic systems using metrics derived from the RNN's connectivity matrix.

Kai: Exactly. They also suggest that analyzing how the network drift decomposes into reversible and irreversible components lets us compare it with mathematical tools like the Helmholtz–Hodge decomposition, which gives a trainable parameter between zero and one.

Mira: That suggests that encoding rotational or cycling dynamics requires more network complexity, which shows up in larger weights in those asymmetric or irreversible parts of the network.

Lev: So ultimately this paper gives us a way to investigate the relationship between models of attractor neural networks and the theory of neural manifolds by quantifying how much complexity is needed for different kinds of dynamics. It’s a framework for comparison.

Kai: It brings us closer to understanding how these complex dynamics emerge from simple network structures by showing that we can map them onto a structured mathematical space.

Mira: That's it for this one on embedding stochastic dynamics in latent manifolds of recurrent neural networks, but the paper also flags some limitations upfront about what it doesn't cover.

Lev: What are those limitations? I need to know what the authors themselves admit they can’t handle with this method.

Kai: They mention that the method focuses on RNNs with low-rank connectivity of the form W = GammaWs, where Gamma is an n by k matrix and k is much smaller than n. That's a key constraint on the network structure.

Mira: And they also note that they are focusing on specific types of dynamics like input-driven attractor switching and stationary diffusions, not necessarily covering every single possible dynamic behavior out there.

Conclusion: Kai: So we're wrapping up on "Embedding stochastic dynamics in latent manifolds of recurrent neural networks." It boils down to how these networks can represent a whole range of nonlinear dynamics given specific drift and diffusion coefficients within a low-dimensional space.

Mira: Yeah, it really shows how the drift–diffusion matching framework lets us train continuous time recurrent neural networks to model any nonlinear stochastic differential equation.

Lev: I still have that nagging thought about the computational load. If we are talking about integrating over time for every single trajectory we want to check, how does that actually run on real hardware?

Kai: They are looking at this kind of work on networks with, say, fifty or sixty four neurons and they show traces from those networks trained to approximate systems like the van der Pol oscillator.

Mira: It proves that when you project those dynamics into the learned subspace, you can actually recover trajectories from the real stochastic system that was originally embedded in that network structure.

Lev: That's a good demonstration of feasibility for smaller systems, but I still wonder about scaling up to systems with more degrees of freedom. Where does the overhead really land when we move beyond those small networks?

Kai: They suggest that encoding rotational or cycling dynamics requires more network complexity, which shows up in larger weights in that asymmetric or irreversible component.

Mira: That decomposition into reversible and irreversible parts lets us compare it to other mathematical tools like the Helmholtz–Hodge decomposition, which gives you a trainable parameter between zero and one.

Kai: It brings us closer to understanding how these complex dynamics emerge from simple network structures, even with just low-dimensional latent spaces.

Mira: That's a solid piece of theory to chew on as we try to crack the neural code.

Lev: I think this work sets up a good benchmark for what kind of complexity we need to expect when trying to model these things in real systems. So that’s everything on this paper.

Ramon Nartallo-Kaluarachchi, Renaud Lambiotte, Alain Goriely

Mathematical Institute, University of Oxford · Centre for Eudaimonia and Human Flourishing, University of Oxford · Complexity Science Hub, Vienna

cond-mat.dis-nn, cond-mat.stat-mech, cs.LG, q-bio.NC

Submitted: 2026-02-16

Updated: 2026-11-06

Comments: 26 pages, 17 figures

Code: https://github.com/rnartallo/ddm

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: Recurrent neural networks can faithfully embed the drift and diffusion of given stochastic differential equations, including nonlinear and nonequilibrium dynamics such as chaotic attractors…

Key concepts

Drift-Diffusion Matching (DDM)
This is the core training method where the RNN is trained to match the specific drift and diffusion coefficients of a target stochastic differential equation. It works by assuming a low-rank connectivity structure, ensuring that the network's dynamics stay tangent to a specific low-dimensional subspace defined by its parameters.
Input-driven Attractor Switching
This capability allows the RNN to use external inputs to decide which stable state (minimum of an energy potential) it should be in. By manipulating these inputs, the system can switch between different memory states or attractors based on sensory information, effectively retrieving specific stored information.
Attractor-cycling Stationary Diffusions
This mechanism models systems that autonomously move between stable states in a preferred direction using irreversible, rotational currents. This allows the network to cycle through related memories or states without external input, mimicking the autonomous transition between different memory configurations.

Terminology

Summary

Recurrent neural networks can faithfully embed the drift and diffusion of given stochastic differential equations, including nonlinear and nonequilibrium dynamics such as chaotic attractors, by training them to represent arbitrary nonlinear SDEs within a low-dimensional latent subspace <ref:2602. p. 0

Where the paper says it

Here we introduce a general framework, which we term drift–diffusion matching, for training continuous-time RNNs to represent arbitrary, nonlinear SDEs, with given drift and diffusion coefficients, within a low-dimensional latent subspace. Allowing asymmetric connectivity, we show that RNNs can faithfully embed the drift and diffusion of a given stochastic differential equation, including nonlinear and nonequilibrium dynamics such as chaotic attractors.

14885#pg2>. This framework extends attractor neural network theory beyond equilibrium by allowing asymmetric connectivity to implement a broad class of dynamical computations within low-dimensional manifolds <ref:2602. p. 0
Where the paper says it

To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results extend attractor neural network theory beyond equilibrium, showing that asymmetric neural populations can implement a broad class of dynamical computations within low-dimensional manifolds, unifying ideas from associative memory, nonequilibrium statistical mechanics, and neural computation.

p. 0
Where the paper says it

To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results extend attractor neural network theory beyond equilibrium, showing that asymmetric neural populations can implement a broad class of dynamical computations within low-dimensional manifolds, unifying ideas from associative memory, nonequilibrium statistical mechanics, and neural computation.

14885#pg4>.

Drift-Diffusion Matching Framework

The core methodology introduced is the drift–diffusion matching (DDM) framework, which trains continuous-time RNNs to represent arbitrary nonlinear SDEs with given drift and diffusion coefficients within a low-dimensional latent subspace <ref:2602.14885#pg4>. The training involves directly matching the drift and diffusion of the target process to the RNN's dynamics in an affine subspace <ref:2602.14885#pg4>. This is achieved by assuming a low-rank connectivity of the form W = ΓWs, I = ΓIs +b, B = ΓBs, which ensures that for any u ∈ A, the drift and diffusion are tangent to the subspace <ref:2602. p. 2

Where the paper says it

As a result, Γ † (u−b) = y for all different u⊥ ⊥ Im(Γ), thus it is not injective. To prevent dynamics in directions that are not ‘visible’ under the projection, we enforce a low-rank parametrisation of the RNN, W = ΓWs , I = ΓIs +b, B = ΓBs , (11) where Ws ∈ R k×n , Is ∈ R k , Bs ∈ R k×d , where we recall that d is the number of independent noise sources.

p. 2
Where the paper says it

As a result, Γ † (u−b) = y for all different u⊥ ⊥ Im(Γ), thus it is not injective. To prevent dynamics in directions that are not ‘visible’ under the projection, we enforce a low-rank parametrisation of the RNN, W = ΓWs , I = ΓIs +b, B = ΓBs , (11) where Ws ∈ R k×n , Is ∈ R k , Bs ∈ R k×d , where we recall that d is the number of independent noise sources.

14885#pg4>. The DDM loss function is defined as LDDM = f(y) +y−Wsh(Γy+b)−Is (14) + λdiffσ2Ik −BsB⊤s, which optimizes over the parameters to align the RNN dynamics with the target SDE <ref:2602. p. 3
Where the paper says it

Given this insight, we introduce the DDM loss function, LDDM = ||f(y) +y−Wsh(Γy+b)−Is || (14) +λdiff||σ 2 Ik −BsB ⊤ s ||, where λdiff is a hyper-parameter controlling the relative importance of the drift and diffusion losses, and where we optimise over the parameters {Γ,b,Ws ,Is ,Bs}, for a total of 2nk +n+k +kd free parameters. Once the two-layer perceptron has been trained, the RNN can be reconstructed from the weights and biases to yield an autonomous neural circuit, in the form of an SDE, from which trajectories can be sampled using a stochastic integrator.

p. 3
Where the paper says it

Given this insight, we introduce the DDM loss function, LDDM = ||f(y) +y−Wsh(Γy+b)−Is || (14) +λdiff||σ 2 Ik −BsB ⊤ s ||, where λdiff is a hyper-parameter controlling the relative importance of the drift and diffusion losses, and where we optimise over the parameters {Γ,b,Ws ,Is ,Bs}, for a total of 2nk +n+k +kd free parameters. Once the two-layer perceptron has been trained, the RNN can be reconstructed from the weights and biases to yield an autonomous neural circuit, in the form of an SDE, from which trajectories can be sampled using a stochastic integrator.

14885#pg4>.

Encoding Diverse Dynamical Behaviors

The framework enables RNNs to encode various types of dynamics, including those that are neither purely gradient flows nor purely time-reversible <ref:2602.14885#pg4>. Two classes of SDEs are introduced:

  1. Input-driven attractor-switching, which uses network inputs to switch between the minima of an energy potential <ref:2602. p. 7

    Where the paper says it

    Based on our method to train an asymmetric RNN to encode a target SDE in a latent affine subspace, we now introduce two classes of SDEs that are defined by energy potential gradients and thus have attracting dynamics. The first of these uses network inputs to switch between the minima of the energy potential.

    p. 7
    Where the paper says it

    Based on our method to train an asymmetric RNN to encode a target SDE in a latent affine subspace, we now introduce two classes of SDEs that are defined by energy potential gradients and thus have attracting dynamics. The first of these uses network inputs to switch between the minima of the energy potential.

    14885#pg4>. This allows the system to retrieve a particular memory based on sensory information or other disjoint neural circuits <ref:2602. p. 7
    Where the paper says it

    Here the network inputs represent sensory information or the output of a disjoint neural circuit, which prompts the RNN to retrieve a particular memory in light of this information. This is a more complex extension of Hopfield’s model, where the initial network state serves as the input, and the system cannot transition between memories.

    14885#pg10>.

  2. Attractor-cycling stationary diffusions, which use irreversible, rotational currents to cycle between minima in a preferred direction <ref:2602. p. 7

    Where the paper says it

    This is a more complex extension of Hopfield’s model, where the initial network state serves as the input, and the system cannot transition between memories. The second is a class of stationary diffusions that use irreversible, rotational currents to cycle between minima in a preferred direction.

    p. 7
    Where the paper says it

    This is a more complex extension of Hopfield’s model, where the initial network state serves as the input, and the system cannot transition between memories. The second is a class of stationary diffusions that use irreversible, rotational currents to cycle between minima in a preferred direction.

    14885#pg4>. This models the ability of a memory system to autonomously transition between related memories <ref:2602. p. 7
    Where the paper says it

    The second is a class of stationary diffusions that use irreversible, rotational currents to cycle between minima in a preferred direction. In this case, our model attempts to describe the ability of a memory system to autonomously transition between a series of a related memories in a preferred direction.

    14885#pg10>.

Decomposition of Network Dynamics

To understand how dynamics are encoded, two decompositions of the RNN are introduced: one based on its asymmetric connectivity and another on its time-irreversibility <ref:2602.14885#pg4>. The symmetric component descends an energy landscape, while the asymmetric component encodes rotational dynamics <ref:2602. p. 3

Where the paper says it

Symmetric-asymmetric decomposition of neural network dynamics The observations made in the previous section, that the projected dynamics of a symmetric RNN remain a generalised gradient flow, motivate a decomposition of a trained RNN into a symmetric and asymmetric component. We consider W = Wsym +Wasym, where the symmetric component, Wsym, descends an energy landscape, whilst the asymmetric component, Wasym, encodes rotational dynamics, ˆf = ˆfsym +ˆfasym, (16) ˆfsym(u) = −u+Wsymh(u) +I, ˆfasym(u) = Wasymh(u).

14885#pg4>. A more motivated decomposition splits the network as W = Γ(omega+Π), where Γomega is symmetric and represents the energy-descending part, and ΓΠ is asymmetric and encodes rotational dynamics <ref:2602. p. 3
Where the paper says it

Symmetric-asymmetric decomposition of neural network dynamics The observations made in the previous section, that the projected dynamics of a symmetric RNN remain a generalised gradient flow, motivate a decomposition of a trained RNN into a symmetric and asymmetric component. We consider W = Wsym +Wasym, where the symmetric component, Wsym, descends an energy landscape, whilst the asymmetric component, Wasym, encodes rotational dynamics, ˆf = ˆfsym +ˆfasym, (16) ˆfsym(u) = −u+Wsymh(u) +I, ˆfasym(u) = Wasymh(u).

14885#pg4>.

Analysis of Latent Dynamics

The analysis reveals that low-rank, symmetric RNNs induce an energy potential over the subspace, precluding limit-cycles or chaotic attractors unless asymmetry is present <ref:2602.14885#pg4>. The decomposition into reversible and irreversible components of the network drift allows for a comparison with the Helmholtz-Hodge decomposition of the latent process, yielding a trainable parameter ρ ∈ [0,1] <ref:2602. p. 7

Where the paper says it

Otherwise, the process is in a nonequilibrium steadystate (NESS), is time-irreversible, and produces entropy. We also introduce the Helmholtz-Hodge decomposition (HHD) of the process, which decomposes the drift of a stationary diffusion into two components, f = frev +firr, frev = D∇logπ, firr = Jss/π, (29) where frev is a time-reversible (shifted) gradient flow and firr is the time-irreversible circulation [50].

14885#pg4>. This suggests that encoding rotational or cycling dynamics requires more network complexity as indicated by larger weights in the irreversible/asymmetric component <ref:2602. p. 1
Where the paper says it

In order to investigate how the dynamics are encoded in the network, we introduce two decompositions of an RNN, the first based on the asymmetry of the connectivity, and the second on the time-irreversibility of the RNN. This theoretical model suggests that populations of neurons with asymmetric connectivity are capable of encoding a wide array of dynamical systems, including attractor-switching and -cycling dynamics, thus offering a mathematical extension of Hopfield’s model which bridges to the theory of neural manifolds.

14885#pg4>.

Reconstruction and Complexity Metrics

The affine subspace can be reconstructed from trajectories using Principal Component Analysis (PCA), which is the maximum-likelihood estimator of the subspace when there is isotropic Gaussian variation orthogonal to it <ref:2602.14885#pg4>. The complexity of encoded chaotic systems can be quantified by metrics derived from the RNN's connectivity matrix, such as the spectral radius, mean participation ratio, and participation ratio of the dominant mode <ref:2602.14885#pg4>. These metrics suggest that certain attractors require more network complexity to be encoded <ref:2602. p. 0

Where the paper says it

As an application, we construct RNN realisations of stochastic systems that transiently explore various attractors through both inputdriven switching and autonomous transitions driven by nonequilibrium currents, which we interpret as models of associative and sequential (episodic) memory. To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility.

14885#pg4>.

Conclusion

The DDM framework provides a simple approach for encoding low-dimensional computational dynamics with specified drift and diffusion coefficients in a high-dimensional neural network <ref:2602.14885#pg4>. It offers a theoretical framework for investigating the relationship between models of attractor neural networks and the theory of neural manifolds <ref:2602. p. 1

Where the paper says it

This theoretical model suggests that populations of neurons with asymmetric connectivity are capable of encoding a wide array of dynamical systems, including attractor-switching and -cycling dynamics, thus offering a mathematical extension of Hopfield’s model which bridges to the theory of neural manifolds. RECURRENT NEURAL NETWORKS WITH TARGET LATENT DYNAMICS A.

14885#pg4>. Ultimately, it brings us a step closer to cracking the neural code <ref:2602. p. 0
Where the paper says it

Practically, we would see that the system descends the landscape monotonically, but more quickly in directions where h ′ (ui) is small. arXiv:2602.

14885#pg4>.

Pruning redundant reasoning tokens

Recurrent neural networks can faithfully embed the drift and diffusion of given stochastic differential equations, including nonlinear and nonequilibrium dynamics such as chaotic attractors, by training them to represent arbitrary nonlinear SDEs within a low-dimensional latent subspace <ref:2602.14885#pg2>. This framework extends attractor neural network theory beyond equilibrium by allowing asymmetric connectivity to implement a broad class of dynamical computations within low-dimensional manifolds <ref:2602.14885#pg4>.

How the training signal is built

The training involves introducing the driftdiffusion matching (DDM) framework, which trains the network to directly align the drift and diffusive terms in an affine subspace to a target SDE with known dynamics <ref:2602.14885#pg4>. This is achieved by assuming a low-rank connectivity of the form W = ΓWs, I = ΓIs +b, B = ΓBs, which ensures that for any u ∈ A, the drift and diffusion are tangent to the subspace <ref:2602.14885#pg4>. The DDM loss function is defined as LDDM = f(y) +y−Wsh(Γy+b)−Is (14) + λdiffσ2Ik −BsB⊤s, which optimizes over the parameters to align the RNN dynamics with the target SDE <ref:2602.14885#pg4>.

Where the gains come from

The framework allows RNNs to encode various types of dynamics, including those that are neither purely gradient flows nor purely time-reversible <ref:2602.14885#pg4>. Two classes of SDEs are introduced: 1. p. 1

Where the paper says it

We illustrate this by embedding nonlinear, and nonequilibrium systems, such as chaotic attractors, within the RNN. Next, to illustrate how the framework can be linked to associative memory, we introduce two classes of SDEs, both of which have a number of attracting states.

p. 1
Where the paper says it

We illustrate this by embedding nonlinear, and nonequilibrium systems, such as chaotic attractors, within the RNN. Next, to illustrate how the framework can be linked to associative memory, we introduce two classes of SDEs, both of which have a number of attracting states.

Input-driven attractor-switching, which uses network inputs to switch between the minima of an energy potential <ref:2602.14885#pg4>. 2. Attractor-cycling stationary diffusions, which use irreversible, rotational currents to cycle between minima in a preferred direction <ref:2602.14885#pg4>.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Drift-Diffusion Matching (DDM) allows training continuous-time Recurrent Neural Networks (RNNs) to represent arbitrary nonlinear Stochastic Differential Equations with given drift and diffusion coefficients within a low-dimensional latent subspace. This enables RNNs to faithfully embed the drift and diffusion of a given stochastic differential equation, including nonlinear and nonequilibrium dynamics such as chaotic attractors.

  2. The improved system can perform input-driven attractor switching by using network inputs to switch between the minima of an energy potential, which prompts the RNN to retrieve a particular memory in light of this information. This is achieved by tilting the potential to prioritise a particular minimum, making it a global minimum, as described by f(y) = -nabla[V(y)-c · y].

  3. The system can perform autonomous cycling between attracting states using stationary diffusions that use irreversible, rotational currents to cycle between minima in a preferred direction. This is constructed by adding a rotational component R(y) to the gradient flow, where R is divergence-free with respect to the stationary density, thus allowing the process to cycle between attractors in a specified order using irreversible currents.

  4. The system can perform attractor-cycling dynamics for complex non-coplanar processes by constructing a rotational field R(y) that pushes trajectories around a differentiable curve γ through the minima, even when those minima are non-coplanar in three dimensions. This is done by defining the rotational vector field R(y) = alpha pi(y) g(y) M nabla phi(y), where M is an antisymmetric matrix.

  5. The system can perform a Helmholtz-Hodge decomposition of the network dynamics, decomposing the drift F as Frev + Firr, allowing for the sampling of trajectories from both components to see how reversible dynamics keep the process near minima while irreversible dynamics show limit-cycle behaviour.

Abstract

Recurrent neural networks (RNNs) provide a theoretical framework for understanding computation in biological neural circuits through their ability to support rich time-dependent behaviour. Whilst high-dimensional by construction, they can be used to encode low-dimensional dynamical systems, for example by restricting to low-rank connectivity, hence mirroring the emergence of latent neural manifolds found in empirical neural data. Here, we exploit a relationship between RNNs and a class of universal approximators to train networks that can represent arbitrary, nonlinear stochastic differential equations (SDEs), with given drift and diffusion coefficients, within a low-dimensional latent subspace. We show that RNNs can faithfully embed the drift and diffusion of a given SDE, including nonlinear and nonequilibrium dynamics such as chaotic attractors. We apply this approach by constructing RNN realisations of stochastic systems that transiently explore various attractors through both input-driven switching and autonomous transitions driven by nonequilibrium currents, which we interpret as models of associative and sequential (episodic) memory. To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results show that neural populations can implement a broad class of stochastic dynamics within low-dimensional manifolds, unifying ideas from the dynamics of neural computation, the theory of neural manifolds, associative memory, and nonequilibrium statistical mechanics.

Sources

Related papers