DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

arXiv:2610.15185 · cs.LG · Submitted 2026-10-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time".

Jane: The gist: DeltaNet-PC develops a predictive coding system that learns to perform test-time verbatim memorization of sequential information without using backpropagation through time.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: To summarize what DeltaNet-PC does, it takes a standard predictive coding model and modifies the energy function to achieve test-time memorization by focusing specifically on that latent variable update mechanism.

Jane: Basically, they use a joint distribution p(k one:L, v one:L,S) restricted to the memory model over a sequence of length L, and the energy function is set up like ridge regression on all key and value pairs for every step > <ref:2610.15185#pg1>.

Lu: The energy function involves that ridge term epsilon twoS 2F plus terms for stored pairs and a value-orthogonality regularizer that keeps values from interfering with each other too, which is detailed in section B <ref:2610.15185#pg1>.three > <ref:2610.15185#pg1>.

Meng: And they derive the update rule for the latent variable DeltaSt by setting the gradient of E t-one(S t-one +) to zero, which results in a recursive least squares update rule > <ref:2610.15185#pg1>.

Lalam: That RLS update is what allows them to get an update like S t = S t-one + g t T (p t-1k t), which is equivalent to the recursive least squares algorithm but without needing an inverse at every time step > <ref:2610.15185#pg1>.

Tom: That efficiency gain with RLS is a key part of the mechanism that makes this approach viable for online learning, unlike methods that might require iterative optimization to settle these latents >

Jane: And they show this works on tasks like memorizing a song where the chorus is followed by different bridges, requiring non-Markovian memory in a single sequence >

Lu: They also address how context uncertainty in multi-episode settings can be resolved through contextual inference, using an energy function like E p(c) = one/two enc(x p) - S L R W k enc(x p-one), p theta + psi(c) squared > <ref:2610.15185#pg1>.

The paper's summary: Tom: The paper highlights a few key improvements right here. First, the use of RLS instead of iterative optimization to settle the latents is a big step because it’s more efficient for online scenarios >

Jane: And they also show how they can resolve context uncertainty in multi-episode settings through this contextual inference mechanism, which is essentially inferring the context latent from observed transitions using that energy function >

Lu: They're also showing that they can learn how to use positional information adaptively by having rotary frequencies learn based on task needs, so sequential recall queries each transition at its own position >

Meng: I'm interested in the part about adaptive positional learning because it suggests that the binding of a written sequence just doesn't exert force, but interference with other occurrences of the same token at different positions moves those frequencies >

Lalam: And they also show that for key-value memory, they can achieve one hundred percent test set accuracy in as few as thirty-two training examples, which is a significant improvement over the one thousand twenty-four needed by DeltaNet >

The paper's improvements: Tom: So to wrap up on DeltaNet-PC, the main implication is that treating matrix-valued updates as latent variables instead of slow weights allows for test-time verbatim memorization, which was a capability previously hard to achieve without backpropagation through time >

Jane: It’s positioned well for online learning and in-memory hardware because of those strict requirements for local learning in space and time and bounded memory in the sequence length >

Lu: The work demonstrates how predictive coding natively provides linear attention machinery, allowing it to be positioned for scenarios where we need rapid context inference from observed transitions >

Meng: From a deployment standpoint, the RLS update rule makes this feasible because you don't need heavy iterative optimization to settle those latents before you can use them in production >

Lalam: And the whole paper is about showing that DeltaNet-PC achieves test-time verbatim memorization of sequential information by treating matrix-valued updates as latent variables, and it sets a new benchmark for this capability.

Conclusion: Tom: So to wrap up on DeltaNet-PC, we’re talking about Test-Time Memory Without Backpropagation Through Time. The main implication is that treating matrix-valued updates as latent variables instead of slow weights allows for test-time verbatim memorization, which was a capability previously hard to achieve without backpropagation through time >

Jane: It’s positioned well for online learning and in-memory hardware because of those strict requirements for local learning in space and time and bounded memory in the sequence length >

Lu: The work demonstrates how predictive coding natively provides linear attention machinery, allowing it to be positioned for scenarios where we need rapid context inference from observed transitions >

Meng: From a deployment standpoint, the RLS update rule makes this feasible because you don't need heavy iterative optimization to settle those latents before you can use them in production >

Lalam: And the whole paper is about showing that DeltaNet-PC achieves test-time verbatim memorization of sequential information by treating matrix-valued updates as latent variables, and it sets a new benchmark for this capability >

Tom: It’s about moving the learning from being a slow parameter update to being an inference problem on these latent variables, which is what makes it potentially runnable in real-world scenarios >

Jane: And they show this architecture can handle within-episode structure tests, multi-episode settings, and key-value binding tests >

Meng: That means if you have a song with a chorus followed by different bridges that are non-Markovian, the model can handle that because of how the memory is structured >

Lalam: They also address how to use positional information adaptively by having rotary frequencies learn based on task needs, so sequential recall queries each transition at its own position >

Tom: That RLS update is what allows them to get an update like S t = S t-one + g t T (p t-1k t), which is equivalent to the recursive least squares algorithm but without needing an inverse at every time step >

Jane: They also show how they can resolve context uncertainty in multi-episode settings through this contextual inference mechanism, which is essentially inferring the context latent from observed transitions using that energy function >

Lu: They're also showing that they can learn how to use positional information adaptively by having rotary frequencies learn based on task needs, so sequential recall queries each transition at its own position >

Meng: I'm interested in the part about adaptive positional learning, because it suggests that the binding of a written sequence just doesn't exert force, but interference with other occurrences of the same token at different positions moves those frequencies >

Lalam: And they also show that for key-value memory, they can achieve one hundred percent test set accuracy in as few as thirty-two training examples, which is a significant improvement over the one thousand twenty-four needed by DeltaNet >

Tom: So to wrap up on DeltaNet-PC: the main implication is that treating matrix-valued updates as latent variables instead of slow weights allows for test-time verbatim memorization, which was a capability previously hard to achieve without backpropagation through time >

Jane: It’s positioned well for online learning and in-memory hardware because of those strict requirements for local learning in space and time and bounded memory in the sequence length >

Lu: The work demonstrates how predictive coding natively provides linear attention machinery, allowing it to be positioned for scenarios where we need rapid context inference from observed transitions >

Meng: From a deployment standpoint, the RLS update rule makes this feasible because you don't need heavy iterative optimization to settle those latents before you can use them in production >

Lalam: And the whole paper is about showing that DeltaNet-PC achieves test-time verbatim memorization of sequential information by treating matrix-valued updates as latent variables, and it sets a new benchmark for this capability >

Tom: It’s about moving the learning from being a slow parameter update to being an inference problem on these latent variables, which is what makes it potentially runnable in real-world scenarios >

Jane: And they show this architecture can handle within-episode structure tests, multi-episode settings, and key-value binding tests >

Todd Morrill, Richard Zemel

Columbia University

cs.LG

Submitted: 2026-10-10

Updated: 2026-10-10

Code: https://github.com/jax-ml/jax

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist: DeltaNet-PC develops a predictive coding system that learns to perform test-time verbatim memorization of sequential information without using backpropagation through time.

Key concepts

Latent Variables
Instead of learning slow weights that represent memory updates, DeltaNet-PC treats the per-step memory update (∆St) as a latent variable. This allows the system to learn how to store information directly during testing, rather than relying on traditional, slow parameter learning methods.
Predictive Coding
The system uses predictive coding principles where memory is modeled through an energy function based on the difference between predicted and actual values. This framework allows the model to infer missing or sequential information by minimizing prediction errors across a sequence.
Recursive Least Squares (RLS)
The update rule for the latent memory variable ∆St is derived as a Recursive Least Squares algorithm. This specific update mechanism is computationally efficient because it avoids needing to compute an inverse at every time step, making the system feasible for online learning scenarios.
Rotary Position Code (RoPE)
DeltaNet-PC uses a rotary position code with learned per-band frequencies to encode the position of keys within a sequence. These learned frequencies allow the system to adaptively use positional information, where interference between tokens at different positions drives the frequency learning.

Summary

The gist: DeltaNet-PC develops a predictive coding system that learns to perform test-time verbatim memorization of sequential information without using backpropagation through time.

How it works

DeltaNet-PC treats the per-step memory update as a latent variable, as opposed to a slow weight to be learned, which is motivated by treating matrix-valued updates as latent variables, as opposed to slowly learned parameters The system connects predictive coding to the linear attention literature through a key insight: once you treat matrix-valued updates ∆St to the memory as a latent variable, as opposed to slowly learned parameters. This allows for test-time memorization of sequential information by treating the per-step memory update ∆St as a latent variable, as opposed to a slow weight to be learned.

The architecture employs a joint distribution p(k1:L, v1:L,S) which is restricted to the memory model over a sequence of length L. The energy function for the memory module is E(S) = − log p(k1:L, v1:L,S) = ϵ2∥S∥2F + X L t=1 1/2 vt − Skt2 + const., which is equivalent to ridge regression on all L pairs (kt, vt). The partial energy function accounts for all observed data up to time t, Et(S) = Et−1(S) + 1/2∥vt − Skt2, which is equivalent to ridge regression on all L pairs (kt, vt).

The update rule for the latent variable ∆St is derived by setting the gradient of Et−1(St−1 + ∆) with respect to ∆ to zero, which gives the update ∆St = rtk⊤t (ktk⊤t + At−1) − 1 (6), where At−1 = Ct−1 + ϵI. This leads to the recursive least squares (RLS) update rule, St = St−1 + gtrt (Pt−1kt)⊤, S0 = 0 (7), which is equivalent to the recursive least squares (RLS) algorithm. This update rule no longer requires an inverse to be computed at every time step, which makes it feasible for online learning scenarios.

Key Components and Interpretations

The energy function is composed of several terms, including the ridge term ϵ2∥S∥2F + const., the stored pairs term from Lemma 1, and the value-orthogonality regularizer Rv t (21). The value-orthogonality regularizer encourages values vt to be orthogonal to prior values vi for i < t, which is detailed in §B.3.

The system utilizes a rotary position code R(k, t) with learned per-band frequencies (RoPE) to represent the position of the key in the sequence. The learned frequencies appear in exactly one energy term, the address prior of the learn pass, λk2∥k − R(Wk enc(x in), p)∥2, which trains Wk and R.

Empirical Results

The system is evaluated on three episodic memory scenarios: Within-episode structure tests the model’s ability to handle non-Markovian dynamics in a single sequence. Multi-episode settings test the system’s ability to memorize and reason about multiple sequences, particularly with highly overlapping subsequences. Key-value binding tests whether the system can retrieve a value when cued with a key.

In single-episode sequential memory, the memory can recall sequences with perfect accuracy when L ≤ D, and scaling D solves this problem. In multi-episode settings, the system achieves 100% retrieval accuracy on a held-out test set with K = 8 episodes of length l = 32 in D = 256. For key-value memory, the system achieves 100% test set accuracy in as few as 32 training examples, whereas DeltaNet needs 1,024.

Limitations and Future Directions

The primary limitation of any predictive coding based system is the need to settle latents, which is typically performed with iterative optimization. The authors mitigate this limitation by using the RLS update rule (Equation 7) to update the memory, which is equivalent to settling ∆St under the energy function in Equation 5, but is more efficient since it doesn’t require iterative optimization. Future work must continue to reduce the number of iterations required to perform inference, and amortized inference (Tschantz et al., 2023) provides a promising direction.

This work demonstrates how predictive coding natively provides linear attention machinery by treating matrix-valued updates as latent variables, as opposed to slowly learned parameters. The system is positioned for online learning scenarios and in-memory hardware implementations due to its strict requirements for local learning in space and time and bounded memory in the sequence length. The authors show that their system can learn to infer context from observed transitions, which is enabled by iterative settling and compute “depth”.

The authors compare their work quantitatively to other common memory rules with and without learning and a modern DeltaNet implementation. They also show mathematically and qualitatively how their work relates to other pivotal works in the linear attention and associative memory literature. The paper concludes that treating matrix-valued updates as latent variables allows for test-time verbatim memorization of sequential information.

The authors wrote the first version of the codebase, which provided a clear scaffold for how to run experiments, how to modify the energy function, etc. AI was also useful for brainstorming solutions to challenges encountered throughout this project. The work is a proof-of-concept for a predictive coding system that learns to perform test-time verbatim memorization of sequential information.

The main text describes the model, the training procedure and the experiments, and the appendices provide all remaining implementation details. Appendix B provides the complete energy function, Appendix C describes the datasets, Appendix D provides implementation details, Appendix E provides hyperparameters, Appendix F provides ablations, and Appendix G provides compute cost. The authors will release their codebase upon publication. The work is supported by funds provided by the National Science Foundation and by DoD OUSD (R&E) under Cooperative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence), and the US National Institutes of Health Grant S10OD028632-01. The authors thank Colin Bredenberg, Blake Richards, Charlotte Volk, and Jeffrey Seely for their helpful discussions and feedback on the manuscript. The work is built on JAX (Bradbury et al., 2018) and Equinox (Kidger & Garcia, 2021), and DeltaNet-PC runs in single precision. The authors run experiments on a combination of a single Nvidia RTX 4090 (24 GB) for development and primarily Nvidia H100 (80GB and 94GB variants) GPUs for multi-run experiments involving a range of configurations and 3 random seeds. The median runtimes are about 10 minutes. The work is built on JAX (Bradbury et al., 2018) and Equinox (Kidger & Garcia, 2021), and DeltaNet-PC runs in single precision.

Improvements for AI systems

  1. The system can perform test-time memorization of sequential information by treating the per-step memory update ∆St as a latent variable, as opposed to a slow weight to be learned. This allows for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.

  2. The system can resolve context uncertainty in multi-episode settings by performing contextual inference, where the context latent is inferred from observed transitions using the energy function to find a hypothesis that best explains the evidence: Ep(c) = 1/2 enc(xp) − SL R Wk enc(xp−1), p θR + ψ(c) squared.

  3. The system can learn how to use positional information adaptively by having rotary frequencies learn based on task needs: Sequential recall queries each transition at its own position, so the binding just written exerts no force. What moves the frequencies is interference with other occurrences of the same token stored at other positions.

  4. The system can handle non-Markovian dynamics in sequences like songs by utilizing a memory structure that accounts for overlapping contexts, as demonstrated by its ability to store multiple sequences of data, like a metro map, and resolve ambiguity in shared blocks through contextual inference.

  5. The system supports key-value retrieval with high accuracy across diverse modalities, achieving 100% test set accuracy on tasks like Omniglot-MNIST binding while maintaining a fixed memory footprint independent of sequence length L.

Abstract

Test-time memorization of sequential information is a foundational capability for any intelligent system that needs to rapidly store and retrieve data, as demonstrated by the attention mechanism in large language models. However, Transformers and linear attention variants are trained via backpropagation through time (BPTT), which has drawbacks such as its enormous memory requirements. Predictive coding (PC), an algorithm that optimizes a prediction-error energy function with local updates, offers a compelling solution but still lacks test-time memory for sequential data. We develop a solution to this problem by treating the per-step memory update ΔS t as a latent variable, as opposed to a slow weight to be learned. By taking the log probability of a standard PC-based probabilistic model, we derive a PC-native solution for verbatim sequence memory, and relate our update rule to common choices in the linear attention literature (e.g., DeltaNet). We present an extensive set of empirical tests to probe our system's properties, including its scaling capacity, memory footprint, and ability to infer a context from evidence (e.g., bank (river) vs. bank (money)). We show how DeltaNet-PC can be used for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.

Sources

Related papers