Understanding Sample Efficiency in Predictive Coding

arXiv:2605.11911 · cs.LG · Submitted 2026-05-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Understanding Sample Efficiency in Predictive Coding".

Jane: Here's the gist of the paper. Predictive Coding (PC) enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive.

Tom: This work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP and can guide how PC should be parametrised to learn most effectively.

Jane: Quantifying Learning Efficiency The authors quantify the efficiency of learning in BP and PC through a metric called “target alignment,” which measures how closely the change in the output of the network is aligned to the output prediction error.

Tom: They derive and empirically validate analytical expressions for target alignment in Deep Linear Networks (DLNs).

Paper summary: Tom: So we've talked about the initial claims. To summarize this paper "Understanding Sample Efficiency in Predictive Coding," the authors introduce a way to measure learning efficiency called target alignment, which compares how closely the network's output changes to its prediction error.

Jane: The central thesis is that predictive coding is more sample efficient than backpropagation because it handles interference much better, especially in deep and narrow networks.

Lu: They build analytical expressions for target alignment specifically within Deep Linear Networks and empirically validate them by showing that learning in predictive coding is more efficient than backpropagation across various network structures.

Meng: It sounds like they've found a way to mathematically show *why* PC is better, not just observe it empirically during training runs.

Lalam: They specifically point out that the interference distortion seen in backpropagation comes from weight updates being affected by correlations between weights and how poorly conditioned those weights are.

Tom: That correlation thing is what usually trips up my training stability when I’m trying to tune a model, so that's a key part of the mechanism they're digging into.

Jane: And they show that this interference is minimized in predictive coding because the learning dynamics approximate natural gradient descent more closely.

Lu: They show that for a hierarchical model with L layers, backpropagation minimizes an energy function involving products of weights from the input to layer L, while predictive coding minimizes a different energy function focused on layer-wise predictions against targets.

Meng: So they're essentially showing two competing mathematical ways to minimize error, and one way is cleaner for the learning process.

Lalam: They conclude that by tuning the learning rates correctly—using those specific scalar values for alpha—predictive coding achieves optimal target alignment when training on an individual input output pair.

Conclusion: Tom: So looking at the whole paper "Understanding Sample Efficiency in Predictive Coding," it seems they’re providing a solid mathematical foundation for why predictive coding is superior to backpropagation in terms of sample efficiency.

Jane: The implication for us is that if we use this layer-specific learning rate approach, we can build models that learn more effectively from fewer data points and without getting bogged down by weight interference.

Lu: It suggests a path forward for designing architectures where the learning process itself is guided by these alignment properties rather than just minimizing a standard squared error energy function.

Meng: From an engineering standpoint, it means we can design training schedules that are more robust, potentially reducing the amount of data you need to get good performance out of a network on real traffic.

Lalam: The paper gives us concrete conditions—those layer-specific learning rates—that guarantee this interference-free learning for single samples and batch learning, which is a very practical result.

Tom: It’s not saying predictive coding is the only way to do things, but it does provide a strong mechanistic understanding of why it works better in these specific contexts.

Jane: The authors give us tools to guide how we should parameterize predictive coding for the best learning outcomes, which is really helpful for anyone trying to apply these ideas.

Nuffield Department of Clinical Neurosciences, University of Oxford · MRC Centre of Research Excellence in Restorative Neural Dynamics

cs.LG

Submitted: 2026-05-12

Updated: 2026-11-10

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 96/100

The gist: Predictive Coding (PC) enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive.

Key concepts

Target Alignment
This is a metric used to measure how well the network's output change matches the prediction error. A high target alignment score indicates that the network is successfully aligning its updates with what it predicts, suggesting more efficient learning.
Interference Mechanisms
In standard BP, weight updates can be distorted by interference caused by correlated or poorly conditioned weights. This interference is expected to be large when the relationship between previous and current layers' weights deviates significantly from being the identity matrix.
Layer-Specific Learning Rates
Using different learning rates for each layer in PC ensures that a single sample can be learned without interference. When these rates are set correctly, PC achieves optimal target alignment even when learning on just one input-output pair.
Generalization to Modern Architectures
The analytical framework developed for simple Deep Linear Networks (DLNs) applies directly to more complex structures like ResNets (networks with skip connections). This confirms that PC's interference-free learning benefits hold true regardless of the network's depth, width, or weight conditioning.

Terminology

Summary

Predictive Coding (PC) enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. p. 0

Where the paper says it

Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive.

This work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP and can guide how PC should be parametrised to learn most effectively. p. 0
Where the paper says it

We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.

Quantifying Learning Efficiency

The authors quantify the efficiency of learning in BP and PC through a metric called “target alignment,” which measures how closely the change in the output of the network is aligned to the output prediction error. They derive and empirically validate analytical expressions for target alignment in Deep Linear Networks (DLNs). p. 0

Where the paper says it

To address this, we quantify the efficiency of learning in BP and PC through a metric called “target alignment”, which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks.

They show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. p. 0
Where the paper says it

We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks.

Interference Mechanisms

The paper shows that learning in PC has less interference than BP, which is especially pronounced in deep, narrow and pre-trained networks. In BP, the weight updates are distorted by parameter-dependent interference arising from correlated and poorly conditioned weights. p. 8

Where the paper says it

In particular, BP updates are distorted by parameter-dependent interference arising from correlated and poorly conditioned weights, whereas PC attenuates interference, providing improved target alignment. We further show that, by introducing layer-dependent learning rates one can guarantee interference-free learning for single samples, independently of the architecture or weight conditioning.

This interference is expected to be large when WL:l−1W⊤L:l−1 deviates substantially from the identity matrix. p. 3
Where the paper says it

Therefore, it only rescales the update magnitude and does not contribute to interference. Interference is therefore expected to be large when WL:l−1W⊤ L:l−1 deviates substantially from the identity matrix.

Guaranteed Optimal Learning

The authors demonstrate that PC with layer-specific learning rates guarantees interference-free single-sample learning, while PC with weight update rescaling factors enables interference-free batch learning. Theorem 1 shows that for layer-specific adaptive learning rates with αl set to the scalar (x∗l−1xˆl−1)−1, PC achieves optimal target alignment when learning on an individual input-output pair. p. 6

Where the paper says it

Theorem 1 (PC with adaptive learning rate). For layer-specific adaptive learning rates with αl set to the scalar (x ∗⊤ l−1xˆl−1) −1 , PC achieves optimal target alignment when learning on an individual input-output pair.

Generalization to Modern Architectures

The analytical framework developed for DLNs extends directly to linear residual networks (ResNets), revealing that PC’s interference-free learning properties remain unchanged under skip connections. The analysis confirms that layer-specific learning rates guarantee perfect alignment in ResNets, independent of network depth, width, or weight conditioning. p. 8

Where the paper says it

We further show that, by introducing layer-dependent learning rates one can guarantee interference-free learning for single samples, independently of the architecture or weight conditioning. Finally, extending the analysis to multi-sample learning reveals an additional source of cross-sample interference.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Target alignment can be maximized by using weight initialisation that preserves activity norm across model layers to ensure higher target alignment than BP is achieved in deep models. This allows for better learning dynamics when weights are poorly conditioned, such as in pre-trained models.

  2. For single sample training, layer specific adaptive learning rates set to alpha l equal to the inverse of the scalar factor x star l minus 1 x hat l minus 1 can guarantee interference free learning irrespective of the architecture or weight conditioning.

  3. For batch training, using weight update rescaling factors Al equal to (1 over B PB b=1 [x star l minus 1, bx hat l minus 1, b]) - 1 can achieve optimal target alignment independently of batch size, mitigating inter-sample interference.

  4. For linear residual networks trained with predictive coding, the introduction of skip connections allows PC to inherit its interference free learning properties while enjoying improved baseline conditioning as a consequence of their structure.

  5. The derived mechanisms allow AI systems to learn more sample efficient by leveraging higher target alignment, leading to longer training trajectories compared to standard backpropagation.

Abstract

Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called ``target alignment'', which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.

Sources

Related papers