Stochastic Thermodynamics of Associative Memory
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.
Kai: We're looking at the paper: "Stochastic Thermodynamics of Associative Memory".
Mira: Here's the gist of the paper. Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their computational properties into the language of dynamical systems and energy landscapes This formulation provides a natural setting for studying thermodynamics and computation in neural systems because DenseAMs are simultaneously simple enough to admit analytic treatment and rich enough to implement nontrivial computational function Here we define the thermodynamic entropy production associated with the operation of such networks, and study polynomial DenseAMs at intermediate memory load Thermodynamic Framework The paper employs stochastic thermodynamics to describe non-equilibrium behavior and energetics of systems evolving under the influence of noise The total (irreversible) entropy production in the bath+network system is then given by: ∆Stot = β(Wt0→tf − ∆F) ≥ 0 This form of irreversible entropy production is what the authors focus on in this paper Equilibrium Analysis of DenseAMs The equilibrium behavior of polynomial DenseAMs is characterized by stationary distributions and equilibrium free energies For quadratic networks at equilibrium in the absence of external fields, the entropy and free energy can be expressed in purely in terms of the alignments at large N, without any microscopic details of the system The local free energy extremum at zero alignment is always concave up, and thus represents a local free energy minimum (Fig.
Kai: 2) This local minimum plays a crucial role in the dynamics of these networks at finite temperature as we will discuss in the next section Dynamical Characterization and Failure Modes The authors establish that dense networks with higher order nonlinearities have a failure mode of pattern completion at nonzero temperatures that is absent from lower order networks and when the temperature is zero They introduce a method that is exact in the mean field limit for calculating the amount of work expended in these networks under arbitrary fast driving The dynamics in the alignments become deterministic, and are given, without the presence of external fields, as: ∂tϕµ = −ϕµ + Ex tanh h kβ(ϕµ)k−1 + kβ Xν̸=µ(ϕν)k−1xνi Tradeoffs in Control Strategies The work done by a particular control strategy u(t) is given by the integral of this quantity Qualitatively, they find that memory reconstruction becomes harder at higher driving frequencies, in the sense that larger error rates γ must be corrected more slowly (smaller ω) then smaller error rates They observe that higher order networks incur a greater work cost when memory recovery is successful for the class of control strategies considered here In general, while higher-order networks have increased storage capacity and better reconstruction accuracy, they incur greater power cost and require stronger control fields They conclude that for successful recall we must drive networks more slowly if they are at higher temperature or if the partial memories are more corrupted They expect this trend to persist under broader control strategies, as the energy landscape of higher-order networks is steeper near memory minima but flatter away from them References
1: D.
Mira: H.
Kai: Ackley, G.
Paper summary: Kai: To recap, this paper "Stochastic Thermodynamics of Associative Memory" explores how we can apply thermodynamic concepts to dense associative memory networks that mix ideas from hopfield networks with transformers and diffusion models.
Mira: The core thesis is that by casting these computational properties into the language of dynamical systems and energy landscapes, we get a natural setting for studying thermodynamics and computation in neural systems.
Lev: They are looking at how these networks operate out of equilibrium, which is where the work gets interesting because most studies have focused on equilibrium or zero temperature cases.
Kai: Specifically, they define the thermodynamic entropy production associated with operating these networks and study polynomial DenseAMs when the memory load is at an intermediate level.
Mira: They use stochastic thermodynamics to describe non-equilibrium behavior and energetics, focusing on that total irreversible entropy production given by delta stoT which must be greater than or equal to zero.
Lev: What they claim is that their simplest instantiations implement associative memory recall without needing extensive training because the patterns can just be stored in the network structure itself.
Kai: They also connect the gradients of DenseAM energy landscapes to the minimization of score functions used in diffusion models, which is a really interesting link to other areas of AI research.
Mira: Furthermore, they draw connections between multi neuron couplings responsible for large information storage capacity and effective theories for networks whose interactions are predominantly pairwise.
Lev: They even suggest that astrocytes, which are non-neuronal cells in the brain, might provide a biological substrate for these types of effective multineuronal couplings similar to those in DenseAMs.
Kai: The paper provides a framework that is simultaneously simple enough for analytic treatment and rich enough to implement nontrivial computational functions.
Mira: It's about setting up this natural setting for understanding the thermodynamic costs inherent in energy-based neural networks.
Conclusion: Kai: So looking at "Stochastic Thermodynamics of Associative Memory" by Rooke, Krotov, Balasubramanian and Wolpert, the authors explore how to measure the actual physical cost of using these dense associative memory networks when they are running.
Mira: They show that for successful recall, we need to drive these networks more slowly if they are at a higher temperature or if the partial memories we start with are more corrupted.
Lev: The implication for hardware is that higher-order networks give you better storage and accuracy, but you have to account for the increased power cost and stronger control fields needed.
Kai: It seems like the trade-off is clear: lower-order networks are more thermodynamically efficient when the memory load isn't too high.
Mira: This suggests that for practical applications of AI in neural systems, we need to balance storage capacity against the thermodynamic cost of operating those networks.
Spencer Rooke, *Dmitry Krotov, *Vijay Balasubramanian, *David Wolpert
David Rittenhouse Laboratory, University of Pennsylvania · Computational Neuroscience Initiative, University of Pennsylvania Institute for Research on Computation at IBM Research · Santa Fe Institute
cond-mat.stat-mech
Submitted: 2026-01-03
Updated: 2026-11-15
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models…
Key concepts
- Stochastic Thermodynamics
- This framework describes how systems behave when they are influenced by random noise. It allows researchers to calculate the total irreversible entropy production ($ΔStot), which quantifies the energy lost or dissipated in a system evolving out of equilibrium, linking microscopic dynamics to macroscopic thermodynamic quantities.
- Entropy Production ($ΔStot)
- This is the total, irreversible entropy production in the bath plus network system. It is defined as $\beta(Wt_0\to t_f - \Delta F)$ and must always be non-negative. It serves as a measure of the thermodynamic cost associated with operating these neural networks under noise.
- Work Cost
- This refers to the amount of work expended by a specific control strategy ($u(t)$) when driving the network. The paper shows that this work cost is related to entropy production, and it increases with driving frequency or when memory corruption is high, indicating a thermodynamic penalty for fast or noisy operations.
Terminology
Summary
Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their computational properties into the language of dynamical systems and energy landscapes This formulation provides a natural setting for studying thermodynamics and computation in neural systems because DenseAMs are simultaneously simple enough to admit analytic treatment and rich enough to implement nontrivial computational function Here we define the thermodynamic entropy production associated with the operation of such networks, and study polynomial DenseAMs at intermediate memory load
Thermodynamic Framework
The paper employs stochastic thermodynamics to describe non-equilibrium behavior and energetics of systems evolving under the influence of noise The total (irreversible) entropy production in the bath+network system is then given by: ∆Stot = β(Wt0→tf − ∆F) ≥ 0 This form of irreversible entropy production is what the authors focus on in this paper
Equilibrium Analysis of DenseAMs
The equilibrium behavior of polynomial DenseAMs is characterized by stationary distributions and equilibrium free energies For quadratic networks at equilibrium in the absence of external fields, the entropy and free energy can be expressed in purely in terms of the alignments at large N, without any microscopic details of the system The local free energy extremum at zero alignment is always concave up, and thus represents a local free energy minimum (Fig. 2) This local minimum plays a crucial role in the dynamics of these networks at finite temperature as we will discuss in the next section
Dynamical Characterization and Failure Modes
The authors establish that dense networks with higher order nonlinearities have a failure mode of pattern completion at nonzero temperatures that is absent from lower order networks and when the temperature is zero They introduce a method that is exact in the mean field limit for calculating the amount of work expended in these networks under arbitrary fast driving The dynamics in the alignments become deterministic, and are given, without the presence of external fields, as: ∂tϕµ = −ϕµ + Ex tanh h kβ(ϕµ)k−1 + kβ Xν̸=µ(ϕν)k−1xνi
Tradeoffs in Control Strategies
The work done by a particular control strategy u(t) is given by the integral of this quantity Qualitatively, they find that memory reconstruction becomes harder at higher driving frequencies, in the sense that larger error rates γ must be corrected more slowly (smaller ω) then smaller error rates They observe that higher order networks incur a greater work cost when memory recovery is successful for the class of control strategies considered here
Conclusion and Future Directions
In general, while higher-order networks have increased storage capacity and better reconstruction accuracy, they incur greater power cost and require stronger control fields They conclude that for successful recall we must drive networks more slowly if they are at higher temperature or if the partial memories are more corrupted They expect this trend to persist under broader control strategies, as the energy landscape of higher-order networks is steeper near memory minima but flatter away from them
References
[1] D. H. Ackley, G. E. Hinton, and T. J. Sejnowski
[2] L. Ambrogioni
[3] D. Amit, H. Gutfreund, and H. Sompolinsky
[4] D. Amit, H. Gutfreund, and H. Sompolinsky
[5] D. Attwell and S. B. Laughlin
[6] V. Balasubramanian
[7] V. Balasubramanian
[8] V. Balasubramanian and M. p. 0 Stochastic Thermodynamics of Associative Memory
Spencer Rooke,1, 2 Dmitry Krotov,3 Vijay Balasubramanian,1, 2, 4, ∗ and David Wolpert4, ∗
1David Rittenhouse Laboratory, University of Pennsylvania, Philadelphia, PA 19104, USA
2Computational Neuroscience Initiative, University of Pennsylvania, Philadelphia, PA 19104, USA
3
IBM Research, Cambridge, MA, USA
4Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA
Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial
Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their
computational properties into the language of dynamical systems and energy landscapes. This
formulation provides a natural setting for studying thermodynamics and computation in neural
systems, because DenseAMs are simultaneously simple enough to admit analytic treatment and
rich enough to implement nontrivial computational function.Where the paper says it
[9] V. Balasubramanian, D. p. 0 Stochastic Thermodynamics of Associative Memory
Spencer Rooke,1, 2 Dmitry Krotov,3 Vijay Balasubramanian,1, 2, 4, ∗ and David Wolpert4, ∗
1David Rittenhouse Laboratory, University of Pennsylvania, Philadelphia, PA 19104, USA
2Computational Neuroscience Initiative, University of Pennsylvania, Philadelphia, PA 19104, USA
3
IBM Research, Cambridge, MA, USA
4Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA
Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial
Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their
computational properties into the language of dynamical systems and energy landscapes. This
formulation provides a natural setting for studying thermodynamics and computation in neural
systems, because DenseAMs are simultaneously simple enough to admit analytic treatment and
rich enough to implement nontrivial computational function.Where the paper says it
[10] A. C. Barato and U. Seifert
[11] C. Bender and S. Orszag
[12] J. Bezanson, A. Edelman, S. Karpinski, and V. B Shah
[13] N. Bleistein and R. Handelsman
[14] T. B Brown, B Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry
[15] H Chaudhry, J Zavatone-Veth D Krotov and C Pehlevan
[16] A Coolen
[17] S Danisch and J Krumbiegel
[18] M Demircigil, J Heusel, M Löwe, S Upgang and F Vermet
[19] M Esposito and C Van den Broeck
[20] E Gardner
[21] E Gardner
[27] J J Hopfield
[44] H Ramsauer, B Schäfl, J Lehner, P Seidl, M Widrich, L Gruber, M Holzleitner T Adler D Kreil M Kopp G Klambauer J Brandstetter S Hochreiter
[45] H Rieger, M Schreckenberg, and J Zittartz
[46] Y Roudi and J Hertz
[47] U Seifert
[48] M Suzuki and R Kubo
[49] R Theriault and D Tantari
[50] C E Tripp, J Perr-Sauer, J Gafur, A Nag, A Purkayastha S Zisman and E A Bensen
[51] C Van den Broeck and M Esposito
[52] A Vaswani, N Shazeer, N Parmar, J Uszkoreit L Jones A Gomez L Kaiser and I Polosukhin
[53] D H Wolpert
[54] D H Wolpert, J Korbel, C W Lynn F Tasnim J A Grochow G Karde¸s J B Aimone V Balasubramanian E De Giuli D Doty et al
Appendix A: MFT for Dense Associative Nets
The partition function in the large N limit approaches the effective action evaluated at the saddles For a single memory case, the free energy can be written purely in terms of phi by inserting phi˜⋆ back into the effective action The solution to this equation always has a solution at phi = 0 representing no alignment
Appendix B: Numerical Details
All numerics were performed using Julia [43], partially through the use of the DifferentialEquations.jl package Visualizations were created with CairoMakie
Figure 6 Analysis
The total work density for the (B,C) k = 2, (E,F) k = 3, and (H,I) k = 4 networks at (B,E,H) β = 1 and (C,F,I) β = 2 under the driving strategy in Eq. 45 The higher order networks consume more power under this strategy Slow driving incurs lower work costs (Fig. p. 1 This allows us to compute
work and power costs associated with arbitrary finite time control protocols. Using these tools, we show that for a
representative family of driving strategies, higher order networks require greater power during operation for equal
operation speed. This allows us to compute
work and power costs associated with arbitrary finite time control protocols. Using these tools, we show that for a
representative family of driving strategies, higher order networks require greater power during operation for equal
operation speed.Where the paper says it
Where the paper says it
Final Assessment
The paper finds tradeoffs between the speed, reconstruction accuracy, and thermodynamic cost of memory recall For successful recall we must drive networks more slowly if they are at higher temperature or if the partial memories are more corrupted While higher-order networks have increased storage capacity and better reconstruction accuracy, they incur greater power cost and require stronger control fields Conversely, lower-order networks are more thermodynamically efficient at low memory loads The entropy production in this case equals the work cost times the inverse temperature
Acknowledgments
This work was supported in part by the NSF and DoD OUSD (R & E) under Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence) While this research was in progress, VB was supported in part by the Eastman Professorship at Balliol College, Oxford Part of this work was done while DK was employed by IBM Research
[1] D. H. Ackley, G. E. Hinton, and T J Sejnowski
[2] L Ambrogioni
[3] D Amit, H Gutfreund, and H Sompolinsky
[4] D Amit, H Gutfreund, and H Sompolinsky
[5] D Attwell and S B Laughlin
[6] V Balasubramanian
[7] V Balasubramanian
[8] V Balasubramanian and M J Berry II
[9] V Balasubramanian, D Kimber, and M J Berry II
[10] A C Barato and U Seifert
[11] C Bender and S Orszag
[12] J Bezanson, A Edelman, S Karpinski, and V B Shah
[13] N Bleistein and R Handelsman
[14] T B Brown, B Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan P Shyam G Sastry A Askell S Agarwal A Herbert-Voss G Krueger T Henighan R Child A Ramesh D M Ziegler J Wu C Winter C Hesse M Chen E Sigler M Litwin S Gray B Chess J Clark C Berner S McCandlish A Radford I Sutskever and D Amodei
[15] H Chaudhry, J Zavatone-Veth D Krotov and C Pehlevan
[16] A Coolen
[17] S Danisch and J Krumbiegel
[18] M Demircigil, J Heusel, M Löwe, S Upgang and F Vermet
[19] M Esposito and C Van den Broeck
[20] E Gardner
[21] E Gardner
[27] J J Hopfield
[44] H Ramsauer, B Schäfl, J Lehner, P Seidl, M Widrich L Gruber M Holzleitner T Adler D Kreil M Kopp G Klambauer J Brandstetter S Hochreiter
[45] H Rieger, M Schreckenberg, and J Zittartz
[46] Y Roudi and J Hertz
[47] U Seifert
[48] M Suzuki and R Kubo
[49] R Theriault and D Tantari
[50] C E Tripp, J Perr-Sauer, J Gafur, A Nag, A Purkayastha S Zisman and E A Bensen
[51] C Van den Broeck and M Esposito
[52] A Vaswani, N Shazeer, N Parmar, J Uszkoreit L Jones A Gomez L Kaiser and I Polosukhin
[53] D H Wolpert
[54] D H Wolpert, J Korbel, C W Lynn F Tasnim J A Grochow G Karde¸s J B Aimone V Balasubramanian E De Giuli D Doty et al
The total work density for the (B,C) k = 2, (E,F) k = 3, and (H,I) k = 4 networks at (B,E,H) β = 1 and (C,F,I) β = 2 under the driving strategy in Eq. 45 The higher order networks consume more power under this strategy Slow driving incurs lower work costs (Fig.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Stochastic Thermodynamics of Associative Memory allows for characterizing out-of-equilibrium properties, work requirements, and memory transition times when driving systems with corrupted memories using dynamical mean field theory to calculate work and entropy production costs exactly in the large system size limit.
-
The improved AI system can accurately predict the trade-offs between entropy production, memory retrieval accuracy, and operation speed for polynomial DenseAM networks by evaluating work cost under arbitrary fast driving protocols.
-
The system can be designed to operate lower order networks at lower temperatures when pattern reconstruction is successful, as higher order networks are found to require greater power during operation for equal operation speed in the regime where memory recovery is successful.
-
The AI system can optimize control strategies for DenseAM networks by evaluating instantaneous power and work using macroscopic states of the system phi(t) and u(t), allowing it to find protocols that minimize thermodynamic costs.
-
The system can perform pattern completion with higher order networks, which exhibit a failure mode of pattern completion at nonzero temperatures that is absent from lower order networks and when the temperature is zero.
Abstract
Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their computational properties into the language of dynamical systems and energy landscapes. This formulation provides a natural setting for studying thermodynamics and computation in neural systems, because DenseAMs are simultaneously simple enough to admit analytic treatment and rich enough to implement nontrivial computational function. Aspects of these networks have been studied at equilibrium and at zero temperature, but the thermodynamic costs associated with their operation out of equilibrium are largely unexplored. Here, we define the thermodynamic entropy production associated with the operation of such networks, and study polynomial DenseAMs at intermediate memory load. At large system sizes and intermediate and low load, we use dynamical mean field theory to characterize out-of-equilibrium properties, work requirements, and memory transition times when driving the system with corrupted memories. We characterize a failure mode of higher order networks not observed at zero temperature. Further, we develop a method for calculating work and power costs in the mean field limit. Finally, we find tradeoffs between entropy production, memory retrieval accuracy, and operation speed.
Sources
- Language Models are Few-Shot Learners
- Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models
- Hierarchical Associative Memory
- Modern Methods in Associative Memory
- Carbon Emissions and Large Neural Network Training
- Memorization to Generalization: Emergence of Diffusion Models from Associative Memory
- Measuring the Energy Consumption and Efficiency of Deep Neural Networks: An Empirical Analysis and Design Recommendations
Related papers
- Emergent long-tail dynamics in driven magnets with dynamical frustration
- Statistical Mechanics of Dynamical System Identification
- Second law of thermodynamics in closed quantum many-body systems
- Chemical potentials from structure factors: II. Charged multi-component mixtures
- Genuine quantum scars in Floquet chaotic many-body systems
- Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective