Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective
summary
The gist
Autoregressive generative models, encompassing architectures like Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba…
In short
The paper develops a general theoretical framework using stochastic thermodynamics to analyze autoregressive generative models (like Transformers and Mamba). It introduces an entropy production measure, decomposed into compression loss and model mismatch, allowing researchers to quantify irreversibility in these non-Markovian systems efficiently using standard Monte Carlo sampling.
Key concepts
- Entropy Production ($\mathcal{S}$)
- This is a measure of irreversibility defined as the KL divergence between the forward path probability and the backward path probability. It quantifies how much information is lost during generation, and it must always be non-negative.
- Decomposition into $L_t$ (Compression Loss) and $M_t$ (Model Mismatch)
- The total entropy production is split into two parts: compression loss ($L_t$), which measures the discarded information when summarizing the future, and model mismatch ($M_t$), which quantifies the cost of reusing the forward emission kernel in reverse. Both components are non-negative.
- Computational Tractability via Sampling
- The framework allows entropy production to be estimated from sampled trajectories without needing complex, exponential calculations over all possible histories. The cost scales at most as $O(T^2)$ for Transformers, making it feasible for real LLM applications.
Terminology used across episodes
This episode discusses
- Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective · Paper Radio
- Stochastic thermodynamics for classical non-Markov jump processes
- Stochastic Thermodynamics of Associative Memory · Paper Radio
- On time-reversibility of linear stochastic models
- Neural Entropy
- Qwen3 Technical Report
The paper
Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective · Read on arXiv
Department of Applied Physics, The University of Tokyo · Quantum-Phase Electronics Center (QPEC), The University of Tokyo · Inamori Research Institute for Science
Autoregressive generative models -- including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba -- all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics. As a proof-of-concept experiment with a large language model (LLM), we evaluate the entropy production for a pre-trained Transformer-based model, GPT-2. We find that the token-level entropy production is dominated by a syntactic artifact, while the sentence-level entropy production tends to be larger for causally ordered than for non-causal text sets. This observation is supported by a re-evaluation with a substantially larger model, Qwen3-4B-Base. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression. We also show that the entropy production decomposes exactly into non-negative per-step contributions in terms of retrospective inference, and each of those terms further splits into information-theoretically meaningful terms: a compression loss and a model mismatch. Our results establish a bridge between stochastic thermodynamics and modern generative models, and provide a starting point for using irreversibility as a quantitative probe of the highly non-Markovian processes generated by models such as LLMs.
DOI: 10.1103/tv38-b23y
Transcript
Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.
Kai: I'm Kai.
Mira: And I'm Mira, here with Lev.
Kai: We're looking at the paper: "Stochastic Thermodynamics for Autoregressive Generative Models".
Mira: Here's the gist of the paper. Autoregressive generative models, encompassing architectures like Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba, generate sequences by sampling each output based on a deterministic summary of the past.
Kai: This process results in genuinely non-Markovian observed processes.
Mira: The authors develop a general theoretical framework rooted in stochastic thermodynamics for this class of architectures and introduce an entropy production measure that can be efficiently estimated from sampled trajectories without incurring exponential overhead, despite the non-Markovian nature of the dynamics.
Kai: The core definition of entropy production (S) is established as the KL divergence between forward and backward path measures: S D KLP to(y 1:T) P from(y T:1) = E P
P to(y 1:T) P from(y T:1): at least 0.
Title and authors: Mira: Now let's look closer at what the authors summarize about "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective." They are setting up a general theory to handle all those autoregressive models, from Transformers to Mamba.
Kai: Basically, they take these models that make predictions by sampling based on a summary of what came before them and they frame their process within the lens of stochastic thermodynamics. This gives them this new way to look at things.
Lev: So, instead of just looking at the output distribution or the loss function, they are focusing on how much information is being produced or dissipated during that generation process itself.
Mira: Exactly. The core idea is defining entropy production as a KL divergence between forward and backward path measures. This lets them get a non-negative number that quantifies this production, and then they show how to break it down into compression loss and model mismatch components.
Kai: That breakdown is key because it separates the information loss from the cost of reusing the model's knowledge in reverse. They’re showing that these two parts are separate things contributing to the overall process.
Lev: It sounds like a way to diagnose *why* a generation process might be irreversible, not just if it is. If compression loss is high, maybe it means we are throwing away important future information too early during sampling.
Mira: That’s one way to think about it. And they've shown that they can compute this using Monte Carlo sampling on trajectories, which keeps the overhead at a manageable O of T squared for Transformers and O of T for architectures like Mamba.
Kai: But I want to circle back to the non-Markovian part. The dynamics are genuinely non-Markovian because you're sampling based on a deterministic summary, not just the immediate past state.
Lev: So, even with this non-Markovian structure, they manage to compute it efficiently from a single sampled trajectory without needing to track the full history of states at every step. That’s something I hadn't expected given how complex those dependencies can get.
Mira: It’s a major result because it shows that you don't have to enumerate all those non-Markovian histories for this calculation, which usually leads to exponential complexity. The framework handles the non-Markovianity in a way that allows for an efficient estimate from the sampled data.
Kai: So, they’re claiming they can probe the structure of these models by looking at this entropy production measure directly on observed trajectories. That seems like a very direct way to test their claims about learned world models.
The paper's summary: Lev: Okay, moving on to the specific improvements the authors suggest for this work. What are the practical ways they think we can make this framework better?
Mira: One big thing is that they provide an analytical expression for the entropy production when the model simplifies down to a linear Gaussian case, which is like when it reduces to what we call Kalman innovation representation.
Kai: In that linear case, they find physical meanings for those loss and mismatch terms. Compression loss is said to correspond to a lossy compression of the future information.
Lev: That makes intuitive sense. And model mismatch corresponds to the cost incurred by reusing the forward emission kernel when you look at it from the backward direction, right?
Mira: That's right. They also found that for block-level coarse-grained entropy production, reversing sentence blocks yields a larger difference for causal texts compared to reversing individual tokens.
Kai: So they are suggesting that comparing token reversal versus sentence block reversal is a more telling way to test causality when you’re looking at LLMs.
Lev: That’s interesting because it suggests that the ordering effects might be more pronounced at a higher level of abstraction, like sentences, than at the level of individual words.
Mira: They also point out that they can compute the block-level difference in entropy production using this same efficient per-trajectory cost we discussed earlier. It keeps things computationally tractable.
Kai: But I do want to hear about what they say is still missing or what their own limitations are regarding this method. What's the caveat?
Lev: They state that while they have a proof-of-concept experiment with GPT-two and Qwen3 4B Base, applying this robustly to much larger models in a way that captures the full picture of irreversibility across longer sequences is still something they are working on <ref:2604.07867#pg2,a proof-of-concept experiment with>.
Mira: So the method works for testing, but scaling it up to capture everything perfectly remains an open challenge. They aren't claiming it’s solved for all future LLMs yet.
Kai: That makes sense. It’s a tool to probe, not a complete solution for understanding every aspect of these models yet.
The paper's improvements: Mira: So, to wrap up on "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective," the paper provides a general theoretical framework based on stochastic thermodynamics for autoregressive models. It introduces an entropy production measure that’s efficiently estimable from sampled trajectories without exponential overhead.
Kai: And they showed that this production breaks down into compression loss and model mismatch, both of which are non-negative, giving us a way to quantify irreversibility.
Lev: Essentially, we get a direct probe for the irreversibility of the real-world process described by the token sequence by looking at these internal representations.
Mira: And they show that when comparing token reversal versus sentence block reversal, reversing sentence blocks gives larger differences for causal texts, which is a more interpretable probe of inter-sentence ordering.
Kai: Overall, this paper gives us a way to compute this entropy production directly from the model itself without needing empirical estimation of long-history conditionals or auxiliary hidden state dynamics.
Lev: It’s a solid method for quantifying causality and irreversibility within the internal representations of large language models, provided we keep in mind that it's currently most robust when applied to smaller sequence lengths.
Mira: That’s the gist of it. They give us a quantitative tool to look inside these generative models and see how they are handling temporal structure, setting the stage for future work on scaling this up.
Kai: We'll keep an eye on this paper as they develop the next steps for making sure this framework is ready for those bigger models.
Conclusion: Kai: So we’ve been looking at this paper, "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective," and they’re proposing a way to measure how much information is being produced when these generative models create sequences.
Mira: Exactly. They take all those models, the Transformers and Mamba ones, and put them into a thermodynamic framework to track entropy production during the generation process.
Lev: What I liked was that they managed to do this using only sampled trajectories instead of needing to track every single state history which would just be impossible for real hardware.
Kai: Right, because that's the non-Markovian part, where the dynamics aren't just about what happened right now but a summary of everything before it. But they still got a computable number without any combinatorial overhead from enumerating all those histories.
Mira: And that decomposition into compression loss and model mismatch is really useful because it tells us if the information loss is coming from throwing away the future or just from how the model reuses its own learned patterns going backward.
Lev: From an engineering point of view, knowing that the per-trajectory cost scales at most like O of T squared for Transformers means we can actually run these measurements on reasonably sized models without needing massive supercomputers for every single test.
Kai: But they also showed that in the linear Gaussian case, when it simplifies to a Kalman representation, you get a clear physical meaning for those loss and mismatch terms. It’s not just some abstract number anymore.
Mira: They pointed out that compression loss is essentially about the information you discard about what comes after, while model mismatch is how much you reuse the forward emission kernel when looking backward.
Lev: That makes sense because it gives us a way to diagnose *why* a generation process might be irreversible, not just if it is. It’s like finding the specific mechanism causing the dissipation of information.
Kai: And I found that they showed that for causal texts, reversing sentence blocks results in a larger difference in entropy production compared to reversing individual tokens. That seems like a more interpretable way to test inter-sentence ordering effects.
Mira: It’s true, because it suggests those larger structural differences are more pronounced at the sentence level than at the token level when dealing with causal models.
Lev: So what this means for someone who just listens to the show is that we have a tool now to look inside these large language models and see how they are handling temporal structure, using something called "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective."
Kai: It’s a solid piece of work showing that we can probe the structure of these models by looking at this entropy production measure directly on observed trajectories.
Mira: They also flagged that while it’s a great way to test, applying it robustly to much larger models in a way that captures the full picture of irreversibility across longer sequences is still something they are working on.
Lev: Exactly. It’s a tool to probe, not a complete solution for understanding every aspect of these models yet.
Kai: So we’ve seen how this framework provides an efficient way to quantify irreversibility in the internal representations of large language models, using trajectory sampling instead of expensive history tracking.
Mira: The decomposition into loss and mismatch gives us physical intuition about what's happening when a model samples its next word based on its past.
Lev: Next time we look at a paper, we’ll see if they can apply these thermodynamic concepts to those closed quantum many-body systems papers we talked about earlier.
More episodes
- 2610.32678-Probing the limits of materials design via a comprehensive crystal database of 14.6 million structures with 140 properties
- 2610.33595-Native-oxide-driven layer asymmetry, two-step switching and exchange bias in Co/Ir/Co trilayers
- 2503.19572-Spin models from nonlinear cellular automata
- 2606.23350-Irreversibility Enhances Quantum-Enhanced Markov-Chain Monte Carlo
- 2608.02744-Two-Parameter Ansatz for the Violation of Eigenstate Thermalization
- 2210.00806-Low- and high-energy localization landscapes for tight-binding Hamiltonians in two-dimensional lattices
- 2409.09625-Selective switch for band-edge alignments in pentagonal CdSeTe monolayer: Atom-valley locking
- 2606.20168-Norms, overlaps and Yangian descendants for the Haldane-Shastry spin chain
- 2512.19503-Strain-induced splitting of the CCDW-NCCDW phase transition in 1T-TaS 2
- 2307.12638-Strong localization of microwaves beyond two dimensions in aperiodic Vogel spirals