Toward a Physics of Deep Learning and Brains

summary

Video file (mp4)

The gist

The gist Deep neural networks and brains both learn and share superficial similarities, allowing for a unified theoretical framework derived from non-equilibrium statistical physics to describe their

In short

The paper applies physics from non-equilibrium statistical mechanics to understand deep learning and brain activity by modeling neuronal avalanches as cascades in neural networks. It finds that deep networks learn best when operating near a 'quasi-critical' state between absorbing and active phases. Maximal susceptibility is a better predictor of learning than being exactly at the critical point, offering a blueprint for engineering better network performance.

Key concepts

Neuronal Avalanches
These are cascades of activity in living brains or deep neural networks, similar to earthquakes. They show scale-free statistics and dynamic fluctuations characteristic of phase transitions. Analyzing their size and duration helps characterize the underlying dynamics of information flow through the network.
Quasi-critical Regime
Deep neural networks are strongly driven by inputs, so they don't reach a true critical point. Instead, they operate in a 'quasi-critical' regime—a tunable neighborhood around criticality. This state is still governed by scaling relations similar to those found at true critical points.
Maximal Susceptibility
This concept identifies the optimal starting point for training networks. It suggests that the region where a network is most sensitive to changes (maximal susceptibility) is a more reliable indicator of successful learning than simply being exactly at the critical point itself.

Terminology used across episodes

This episode discusses

The paper

Toward a Physics of Deep Learning and Brains · Read on arXiv

Center for Complex Networks and Systems Research, Indiana University Bloomington Department of Physics Indiana University Institute for Advanced Study Princeton

Deep neural networks and brains both learn and share superficial similarities: processing nodes are likened to neurons and adjustable weights are likened to modifiable synapses. But can a unified theoretical framework be found to underlie them both? Here we show that the equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks. These equations are derived from non-equilibrium statistical physics and show that deep neural networks learn best when poised between absorbing and active phases. Because these networks are strongly driven by inputs, however, they do not operate at a true critical point but within a quasi-critical regime -- one that still approximately satisfies crackling noise scaling relations. By training networks with different initializations, we show that maximal susceptibility is a more reliable predictor of learning than proximity to the critical point itself. This provides a blueprint for engineering improved network performance. Finally, using finite-size scaling we identify distinct universality classes, including Barkhausen noise and directed percolation. This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Toward a Physics of Deep Learning and Brains".

Jane: The gist Deep neural networks and brains both learn and share superficial similarities, allowing for a unified theoretical framework derived from non-equilibrium statistical physics to describe their activity.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We've talked about how this paper connects neuronal avalanches to deep learning dynamics, focusing on that idea of being poised between absorbing and active phases #pg1

Jane: Exactly. The core thesis is that the equations used for neuronal avalanches can be applied to cascades in deep neural networks, derived from non-equilibrium statistical physics #pg1

Lu: They are showing this framework demonstrates learning occurs best when networks are in this specific state between absorbing and active phases #pg1

Meng: I’m still wondering about the practical side. The abstract suggests this is a theoretical connection, so what does it actually *do* for training? #pg1

Tom: It provides a blueprint for engineering improved network performance by using maximal susceptibility as a predictor of learning instead of just proximity to the critical point #pg1

Jane: That’s really concrete. So, if we follow this idea, we're not just guessing where the best training spot is, but we have a measurable quantity—susceptibility—that guides us there #pg1

Lu: They also identify two types of criticality: the edge of chaos and the Widom-like line #pg2

Meng: What’s the difference between those two? Is one better for training than the other? #pg2

Tom: The paper links performance loss to that Widom-like line, which fades as a parameter, sigma 2b, grows #pg2

Jane: So theWidom-like line shows where performance starts dropping as something changes in the network structure or parameters #pg2

Lu: And they also show that trainability overlaps with a quasi-critical plateau #pg1 This suggests there's an area of high susceptibility where learning happens #pg1

Meng: So, this means we can try to tune our network parameters to land us in that high-susceptibility zone, even if it’s not the absolute critical point #pg1

Tom: Precisely. The paper shows that successful training overlaps with this area of heightened susceptibility when sigma 2b increases and shifts to larger values of sigma 2w #pg1

Jane: It sounds like they are giving us a way to understand *why* some architectures learn better than others, based on these underlying physical properties #pg1

Lu: And they conclude that learning tasks require certain levels of proximity to criticality because trainability and learnability start happening around the critical value sigma 2w equals one #pg2 <ref:2509.22649#pg1>

Meng: So, it’s not about finding a magic number; it’s about finding the right *region* of the parameter space where things are physically conducive to learning #pg1

Tom: It's really giving us operational diagnostics. The crackling toolkit lets pipelines track mismatch between exponent relations and distance from criticality #pg1

Jane: That means we can start using standard knobs like dropout or batch normalization not just as general regularization tools, but as ways to steer the training towards this quasi-critical plateau #pg1

Lu: And the final point is that universal features are shared across both biological and artificial systems through classes like Barkhausen noise and directed percolation #pg1

Meng: So, it’s unifying neuroscience and AI under a single physical description, which makes sense for understanding the fundamental limits of what these systems can achieve #pg1

Conclusion: Tom: So wrapping up this paper "Toward a Physics of Deep Learning and Brains," the main point is that we don't need perfect criticality to get good deep learning performance #pg1

Jane: Right. The authors argue that the key is being in this quasi-critical regime, poised between absorbing and active phases, which they characterize by maximal susceptibility rather than just being at a single critical value #pg1

Lu: They’ve established that universal features are shared across both biological and artificial neural networks through specific universality classes like Barkhausen noise and directed percolation #pg1

Meng: For those of us building models for production, this offers a physical understanding of *why* some architectures perform better than others based on these shared laws #pg1

Tom: It gives us a way to engineer better performance by targeting that quasi-critical plateau, using tools like dropout to bias training toward it #pg1

Jane: So the big picture is that both brains and deep learning systems use cascades of activity for information processing, and they follow shared physical rules in how they manage that activity #pg1

More episodes

← Home