Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective

arXiv:2604.07867 · cond-mat.stat-mech, cond-mat.dis-nn · Submitted 2026-04-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Condensed Matter. The day in condensed matter: materials, phases and the measurements under them.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: We're looking at the paper: "Stochastic Thermodynamics for Autoregressive Generative Models".

Mira: Here's the gist of the paper. Autoregressive generative models, encompassing architectures like Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba, generate sequences by sampling each output based on a deterministic summary of the past.

Kai: This process results in genuinely non-Markovian observed processes.

Mira: The authors develop a general theoretical framework rooted in stochastic thermodynamics for this class of architectures and introduce an entropy production measure that can be efficiently estimated from sampled trajectories without incurring exponential overhead, despite the non-Markovian nature of the dynamics.

Kai: The core definition of entropy production (S) is established as the KL divergence between forward and backward path measures: S D KLP to(y 1:T) P from(y T:1) = E P

P to(y 1:T) P from(y T:1): at least 0.

Title and authors: Mira: Now let's look closer at what the authors summarize about "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective." They are setting up a general theory to handle all those autoregressive models, from Transformers to Mamba.

Kai: Basically, they take these models that make predictions by sampling based on a summary of what came before them and they frame their process within the lens of stochastic thermodynamics. This gives them this new way to look at things.

Lev: So, instead of just looking at the output distribution or the loss function, they are focusing on how much information is being produced or dissipated during that generation process itself.

Mira: Exactly. The core idea is defining entropy production as a KL divergence between forward and backward path measures. This lets them get a non-negative number that quantifies this production, and then they show how to break it down into compression loss and model mismatch components.

Kai: That breakdown is key because it separates the information loss from the cost of reusing the model's knowledge in reverse. They’re showing that these two parts are separate things contributing to the overall process.

Lev: It sounds like a way to diagnose *why* a generation process might be irreversible, not just if it is. If compression loss is high, maybe it means we are throwing away important future information too early during sampling.

Mira: That’s one way to think about it. And they've shown that they can compute this using Monte Carlo sampling on trajectories, which keeps the overhead at a manageable O of T squared for Transformers and O of T for architectures like Mamba.

Kai: But I want to circle back to the non-Markovian part. The dynamics are genuinely non-Markovian because you're sampling based on a deterministic summary, not just the immediate past state.

Lev: So, even with this non-Markovian structure, they manage to compute it efficiently from a single sampled trajectory without needing to track the full history of states at every step. That’s something I hadn't expected given how complex those dependencies can get.

Mira: It’s a major result because it shows that you don't have to enumerate all those non-Markovian histories for this calculation, which usually leads to exponential complexity. The framework handles the non-Markovianity in a way that allows for an efficient estimate from the sampled data.

Kai: So, they’re claiming they can probe the structure of these models by looking at this entropy production measure directly on observed trajectories. That seems like a very direct way to test their claims about learned world models.

The paper's summary: Lev: Okay, moving on to the specific improvements the authors suggest for this work. What are the practical ways they think we can make this framework better?

Mira: One big thing is that they provide an analytical expression for the entropy production when the model simplifies down to a linear Gaussian case, which is like when it reduces to what we call Kalman innovation representation.

Kai: In that linear case, they find physical meanings for those loss and mismatch terms. Compression loss is said to correspond to a lossy compression of the future information.

Lev: That makes intuitive sense. And model mismatch corresponds to the cost incurred by reusing the forward emission kernel when you look at it from the backward direction, right?

Mira: That's right. They also found that for block-level coarse-grained entropy production, reversing sentence blocks yields a larger difference for causal texts compared to reversing individual tokens.

Kai: So they are suggesting that comparing token reversal versus sentence block reversal is a more telling way to test causality when you’re looking at LLMs.

Lev: That’s interesting because it suggests that the ordering effects might be more pronounced at a higher level of abstraction, like sentences, than at the level of individual words.

Mira: They also point out that they can compute the block-level difference in entropy production using this same efficient per-trajectory cost we discussed earlier. It keeps things computationally tractable.

Kai: But I do want to hear about what they say is still missing or what their own limitations are regarding this method. What's the caveat?

Lev: They state that while they have a proof-of-concept experiment with GPT-two and Qwen3 4B Base, applying this robustly to much larger models in a way that captures the full picture of irreversibility across longer sequences is still something they are working on <ref:2604.07867#pg2,a proof-of-concept experiment with>.

Mira: So the method works for testing, but scaling it up to capture everything perfectly remains an open challenge. They aren't claiming it’s solved for all future LLMs yet.

Kai: That makes sense. It’s a tool to probe, not a complete solution for understanding every aspect of these models yet.

The paper's improvements: Mira: So, to wrap up on "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective," the paper provides a general theoretical framework based on stochastic thermodynamics for autoregressive models. It introduces an entropy production measure that’s efficiently estimable from sampled trajectories without exponential overhead.

Kai: And they showed that this production breaks down into compression loss and model mismatch, both of which are non-negative, giving us a way to quantify irreversibility.

Lev: Essentially, we get a direct probe for the irreversibility of the real-world process described by the token sequence by looking at these internal representations.

Mira: And they show that when comparing token reversal versus sentence block reversal, reversing sentence blocks gives larger differences for causal texts, which is a more interpretable probe of inter-sentence ordering.

Kai: Overall, this paper gives us a way to compute this entropy production directly from the model itself without needing empirical estimation of long-history conditionals or auxiliary hidden state dynamics.

Lev: It’s a solid method for quantifying causality and irreversibility within the internal representations of large language models, provided we keep in mind that it's currently most robust when applied to smaller sequence lengths.

Mira: That’s the gist of it. They give us a quantitative tool to look inside these generative models and see how they are handling temporal structure, setting the stage for future work on scaling this up.

Kai: We'll keep an eye on this paper as they develop the next steps for making sure this framework is ready for those bigger models.

Conclusion: Kai: So we’ve been looking at this paper, "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective," and they’re proposing a way to measure how much information is being produced when these generative models create sequences.

Mira: Exactly. They take all those models, the Transformers and Mamba ones, and put them into a thermodynamic framework to track entropy production during the generation process.

Lev: What I liked was that they managed to do this using only sampled trajectories instead of needing to track every single state history which would just be impossible for real hardware.

Kai: Right, because that's the non-Markovian part, where the dynamics aren't just about what happened right now but a summary of everything before it. But they still got a computable number without any combinatorial overhead from enumerating all those histories.

Mira: And that decomposition into compression loss and model mismatch is really useful because it tells us if the information loss is coming from throwing away the future or just from how the model reuses its own learned patterns going backward.

Lev: From an engineering point of view, knowing that the per-trajectory cost scales at most like O of T squared for Transformers means we can actually run these measurements on reasonably sized models without needing massive supercomputers for every single test.

Kai: But they also showed that in the linear Gaussian case, when it simplifies to a Kalman representation, you get a clear physical meaning for those loss and mismatch terms. It’s not just some abstract number anymore.

Mira: They pointed out that compression loss is essentially about the information you discard about what comes after, while model mismatch is how much you reuse the forward emission kernel when looking backward.

Lev: That makes sense because it gives us a way to diagnose *why* a generation process might be irreversible, not just if it is. It’s like finding the specific mechanism causing the dissipation of information.

Kai: And I found that they showed that for causal texts, reversing sentence blocks results in a larger difference in entropy production compared to reversing individual tokens. That seems like a more interpretable way to test inter-sentence ordering effects.

Mira: It’s true, because it suggests those larger structural differences are more pronounced at the sentence level than at the token level when dealing with causal models.

Lev: So what this means for someone who just listens to the show is that we have a tool now to look inside these large language models and see how they are handling temporal structure, using something called "Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective."

Kai: It’s a solid piece of work showing that we can probe the structure of these models by looking at this entropy production measure directly on observed trajectories.

Mira: They also flagged that while it’s a great way to test, applying it robustly to much larger models in a way that captures the full picture of irreversibility across longer sequences is still something they are working on.

Lev: Exactly. It’s a tool to probe, not a complete solution for understanding every aspect of these models yet.

Kai: So we’ve seen how this framework provides an efficient way to quantify irreversibility in the internal representations of large language models, using trajectory sampling instead of expensive history tracking.

Mira: The decomposition into loss and mismatch gives us physical intuition about what's happening when a model samples its next word based on its past.

Lev: Next time we look at a paper, we’ll see if they can apply these thermodynamic concepts to those closed quantum many-body systems papers we talked about earlier.

Department of Applied Physics, The University of Tokyo · Quantum-Phase Electronics Center (QPEC), The University of Tokyo · Inamori Research Institute for Science

cond-mat.stat-mech, cond-mat.dis-nn

Submitted: 2026-04-09

Updated: 2026-10-01

Comments: 30 pages, 11 figures

Journal ref: PRX Intelligence 1, 023013 (2026)

DOI: 10.1103/tv38-b23y

Code: https://github.com/openai/gpt-2

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 97/100

The gist: Autoregressive generative models, encompassing architectures like Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba…

Key concepts

Entropy Production ($\mathcal{S}$)
This is a measure of irreversibility defined as the KL divergence between the forward path probability and the backward path probability. It quantifies how much information is lost during generation, and it must always be non-negative.
Decomposition into $L_t$ (Compression Loss) and $M_t$ (Model Mismatch)
The total entropy production is split into two parts: compression loss ($L_t$), which measures the discarded information when summarizing the future, and model mismatch ($M_t$), which quantifies the cost of reusing the forward emission kernel in reverse. Both components are non-negative.
Computational Tractability via Sampling
The framework allows entropy production to be estimated from sampled trajectories without needing complex, exponential calculations over all possible histories. The cost scales at most as $O(T^2)$ for Transformers, making it feasible for real LLM applications.

Terminology

Summary

Autoregressive generative models, encompassing architectures like Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba, generate sequences by sampling each output based on a deterministic summary of the past. p. 0

Where the paper says it

Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective Takahiro Sagawa1, 2, 3 1Department of Applied Physics, The University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan 2Quantum-Phase Electronics Center (QPEC), The University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan 3 Inamori Research Institute for Science (InaRIS), Kyoto-shi, Kyoto 600-8411, Japan (Dated: October 2, 2026) Autoregressive generative models — including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba — all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics.

This process results in genuinely non-Markovian observed processes. p. 0
Where the paper says it

Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective Takahiro Sagawa1, 2, 3 1Department of Applied Physics, The University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan 2Quantum-Phase Electronics Center (QPEC), The University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan 3 Inamori Research Institute for Science (InaRIS), Kyoto-shi, Kyoto 600-8411, Japan (Dated: October 2, 2026) Autoregressive generative models — including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba — all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics.

The authors develop a general theoretical framework rooted in stochastic thermodynamics for this class of architectures and introduce an entropy production measure that can be efficiently estimated from sampled trajectories without incurring exponential overhead, despite the non-Markovian nature of the dynamics. p. 0
Where the paper says it

Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective Takahiro Sagawa1, 2, 3 1Department of Applied Physics, The University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan 2Quantum-Phase Electronics Center (QPEC), The University of Tokyo, Bunkyo-ku, Tokyo 113-8656, Japan 3 Inamori Research Institute for Science (InaRIS), Kyoto-shi, Kyoto 600-8411, Japan (Dated: October 2, 2026) Autoregressive generative models — including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba — all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics.

The core definition of entropy production (S) is established as the KL divergence between forward and backward path measures: S D KLP to(y 1:T) P from(y T:1) = E P[P to(y 1:T) P from(y T:1)] at least 0. p. 0

Where the paper says it

This class encompasses diverse architectures as described above; we bring them under a single framework (Table I) and develop the stochastic thermodynamics of the resulting non-Markovian observed process in a unified manner, by constructing a backward process with the same architectural components applied in reversed temporal order. Conceptually, our definition of the entropy production builds on a line of research in which the KL divergence between forward and backward path measures serves as an operational characterization of irreversibility for the observed process [6–14].

A key finding is the decomposition of this entropy production into non-negative per-step contributions, delta t, where D t E y t+1:T about P to(y t y t+1:T) p t(y t g from t+1(y T:t+1)) at least 0. p. 1
Where the paper says it

We derive an analytical expression for the entropy production, which is numerically verified by applying the Monte Carlo sampling procedure. Furthermore, we show that the entropy production Sy admits an exact retrospective decomposition into a sum of non-negative per-step contributions Dt ≥ 0, each measuring how well the backward model retrodicts the present observation yt from the future.

This decomposition further splits into compression loss (L t) and model mismatch (M t), both of which are non-negative. p. 1
Where the paper says it

Furthermore, we show that the entropy production Sy admits an exact retrospective decomposition into a sum of non-negative per-step contributions Dt ≥ 0, each measuring how well the backward model retrodicts the present observation yt from the future. Each Dt further splits into a compression loss Lt, arising when the backward latent state is a lossy summary of the future, and a model mismatch Mt, arising because the emission kernel designed for forward prediction is reused in the backward direction.

The framework is shown to be computationally tractable using standard Monte Carlo sampling, with a per-trajectory cost of C 1 = 2C LL, where C LL is the cost of a single log-likelihood evaluation. p. 8

Where the paper says it

Since σ requires one forward and one backward pass, and each pass is a log-likelihood evaluation, the per-trajectory cost is C1 = 2CLL, (26) where CLL denotes the cost of a single log-likelihood evaluation for the architecture in question. Roughly speaking, the cost CLL of a single loglikelihood evaluation scales at most as O(T 2 ) for Transformers [38] and as O(T) for recursive architectures such as RNNs [42, 43] and state space models and Mamba [47– 49]; in particular, no combinatorial overhead from enumerating non-Markovian histories enters either factor in (25), the only T-dependence of N being the Monte Carlo factor Var(σ).

This cost scales at most as O(T 2) for Transformers and O(T) for recursive architectures like RNNs, state space models, and Mamba, crucially avoiding combinatorial overhead from enumerating non-Markovian histories. p. 8
Where the paper says it

Since σ requires one forward and one backward pass, and each pass is a log-likelihood evaluation, the per-trajectory cost is C1 = 2CLL, (26) where CLL denotes the cost of a single log-likelihood evaluation for the architecture in question. Roughly speaking, the cost CLL of a single loglikelihood evaluation scales at most as O(T 2 ) for Transformers [38] and as O(T) for recursive architectures such as RNNs [42, 43] and state space models and Mamba [47– 49]; in particular, no combinatorial overhead from enumerating non-Markovian histories enters either factor in (25), the only T-dependence of N being the Monte Carlo factor Var(σ).

The block-level coarse-grained entropy production can also be evaluated at this same per-trajectory cost. p. 9
Where the paper says it

10 The coarse-grained stochastic entropy production for a single trajectory y1:T ∼ P→ is then σ ′ (y1:T ) ≡ ln P(y1:T ) − ln P(˜y ′ 1:T˜ ), (32) which is the difference in log-likelihood between the original text and its block-reversed version, both evaluated by the same model in a single forward pass each. Since the block-reversal map is a bijection on Y T , P ′ ← is a normalized distribution on y1:T .

In the linear Gaussian case—where the model reduces to the Kalman innovation representation—an analytical expression for the entropy production is derived. p. 0

Where the paper says it

This observation is supported by a re-evaluation with a substantially larger model, Qwen3-4B-Base. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression.

The decomposition reveals that L t and M t have physical interpretations: L t corresponds to lossy compression of the future, while M t arises from the reuse of the forward emission kernel. p. 1
Where the paper says it

Each Dt further splits into a compression loss Lt, arising when the backward latent state is a lossy summary of the future, and a model mismatch Mt, arising because the emission kernel designed for forward prediction is reused in the backward direction. The decomposition is formally reminiscent of the evidence lower bound (ELBO) decompositions used in variational inference [52–54], but arises from a fundamentally different starting point (namely, time reversal and entropy production).

Furthermore, the block-level difference in entropy production is consistently larger for causal texts when comparing token reversal versus sentence-block reversal. p. 1
Where the paper says it

As a proof-of-concept experiment with a pre-trained LLM, we evaluate the stochastic entropy production for GPT-2. We show that the token-level entropy production is dominated by a syntactic artifact of token-order reversal, whereas the block-level coarse-grained entropy production, which reverses the order of sentences rather than individual tokens, tends to be larger for causally ordered than for non-causal texts.

The framework demonstrates that S can be computed directly from the model itself without requiring empirical estimation of long-history conditionals or introducing auxiliary stochastic hidden-state dynamics. p. 6
Where the paper says it

As a consequence, Sy is directly computable from the model itself without empirical estimation of longhistory conditionals, and without introducing an auxiliary stochastic hidden-state dynamics. Because our framework does not presuppose any thermal reservoir, we impose no local detailed balance condition.

This approach establishes a quantitative probe for irreversibility and causality within the internal representations of large language models (LLMs). p. 1
Where the paper says it

LLMs are often argued to acquire internal representations of the real world, including spatial and temporal structure — a capacity sometimes discussed in terms of a learned world model [50, 51]. From this perspective, the entropy production of an LLM ultimately aims to probe the irreversibility of the real-world process described by the token sequence — ideally, the temporal or causal ordering of the events narrated in the text.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective provides a general theoretical framework based on stochastic thermodynamics for autoregressive models including Transformers, RNNs, Kalman filters, state space models, and Mamba. This allows the estimation of entropy production from sampled trajectories without exponential overhead despite the non-Markovian nature of the observed dynamics.

  2. Entropy Production Decomposition into Compression Loss and Model Mismatch enables a more detailed understanding of irreversibility. This decomposition splits each per-step contribution Dt into Lt (compression loss, quantifying retrospective information discarded by the backward summary) and Mt (model mismatch, quantifying the cost of reusing the forward emission kernel in the backward direction).

  3. The framework allows for trajectory-level quantification of irreversibility using Monte Carlo sampling without combinatorial overhead from enumerating histories. The per-trajectory cost is written as Ctotal = NC1, where C1 is the cost of evaluating sigma on a single trajectory, which scales at most as O(T 2) for Transformers.

  4. Temporal Coarse-Graining enables the assessment of block-level entropy production, which reverses the order of blocks rather than individual tokens. The coarse-grained stochastic entropy production sigma prime is defined as σ prime (y1:T) = ln P(y1:T) - ln P(y'1:T˜), which is non-negative as a KL divergence.

  5. The framework enables the comparison of causal and non-causal ordering effects between different input text generators, showing that block reversal is consistently larger for causal texts, which may serve as a more interpretable probe of inter-sentence ordering.

Abstract

Autoregressive generative models -- including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba -- all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics. As a proof-of-concept experiment with a large language model (LLM), we evaluate the entropy production for a pre-trained Transformer-based model, GPT-2. We find that the token-level entropy production is dominated by a syntactic artifact, while the sentence-level entropy production tends to be larger for causally ordered than for non-causal text sets. This observation is supported by a re-evaluation with a substantially larger model, Qwen3-4B-Base. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression. We also show that the entropy production decomposes exactly into non-negative per-step contributions in terms of retrospective inference, and each of those terms further splits into information-theoretically meaningful terms: a compression loss and a model mismatch. Our results establish a bridge between stochastic thermodynamics and modern generative models, and provide a starting point for using irreversibility as a quantitative probe of the highly non-Markovian processes generated by models such as LLMs.

Sources

Related papers