Understanding Sample Efficiency in Predictive Coding
summary
The gist
Predictive Coding (PC) enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive.
In short
This work compares Predictive Coding (PC) and Backpropagation (BP) to show that PC learns more efficiently, especially in deep, narrow networks. It uses a metric called 'target alignment' to quantify this efficiency and reveals that PC suffers less interference during learning. The findings suggest how to parameterize PC for optimal learning.
Key concepts
- Target Alignment
- This is a metric used to measure how well the network's output change matches the prediction error. A high target alignment score indicates that the network is successfully aligning its updates with what it predicts, suggesting more efficient learning.
- Interference Mechanisms
- In standard BP, weight updates can be distorted by interference caused by correlated or poorly conditioned weights. This interference is expected to be large when the relationship between previous and current layers' weights deviates significantly from being the identity matrix.
- Layer-Specific Learning Rates
- Using different learning rates for each layer in PC ensures that a single sample can be learned without interference. When these rates are set correctly, PC achieves optimal target alignment even when learning on just one input-output pair.
- Generalization to Modern Architectures
- The analytical framework developed for simple Deep Linear Networks (DLNs) applies directly to more complex structures like ResNets (networks with skip connections). This confirms that PC's interference-free learning benefits hold true regardless of the network's depth, width, or weight conditioning.
Terminology used across episodes
This episode discusses
- Understanding Sample Efficiency in Predictive Coding · Paper Radio
- ePC: Fast and Deep Predictive Coding in Digital Simulation
The paper
Understanding Sample Efficiency in Predictive Coding · Read on arXiv
Nuffield Department of Clinical Neurosciences, University of Oxford · MRC Centre of Research Excellence in Restorative Neural Dynamics
Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called ``target alignment'', which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Understanding Sample Efficiency in Predictive Coding".
Jane: Here's the gist of the paper. Predictive Coding (PC) enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive.
Tom: This work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP and can guide how PC should be parametrised to learn most effectively.
Jane: Quantifying Learning Efficiency The authors quantify the efficiency of learning in BP and PC through a metric called “target alignment,” which measures how closely the change in the output of the network is aligned to the output prediction error.
Tom: They derive and empirically validate analytical expressions for target alignment in Deep Linear Networks (DLNs).
Paper summary: Tom: So we've talked about the initial claims. To summarize this paper "Understanding Sample Efficiency in Predictive Coding," the authors introduce a way to measure learning efficiency called target alignment, which compares how closely the network's output changes to its prediction error.
Jane: The central thesis is that predictive coding is more sample efficient than backpropagation because it handles interference much better, especially in deep and narrow networks.
Lu: They build analytical expressions for target alignment specifically within Deep Linear Networks and empirically validate them by showing that learning in predictive coding is more efficient than backpropagation across various network structures.
Meng: It sounds like they've found a way to mathematically show *why* PC is better, not just observe it empirically during training runs.
Lalam: They specifically point out that the interference distortion seen in backpropagation comes from weight updates being affected by correlations between weights and how poorly conditioned those weights are.
Tom: That correlation thing is what usually trips up my training stability when I’m trying to tune a model, so that's a key part of the mechanism they're digging into.
Jane: And they show that this interference is minimized in predictive coding because the learning dynamics approximate natural gradient descent more closely.
Lu: They show that for a hierarchical model with L layers, backpropagation minimizes an energy function involving products of weights from the input to layer L, while predictive coding minimizes a different energy function focused on layer-wise predictions against targets.
Meng: So they're essentially showing two competing mathematical ways to minimize error, and one way is cleaner for the learning process.
Lalam: They conclude that by tuning the learning rates correctly—using those specific scalar values for alpha—predictive coding achieves optimal target alignment when training on an individual input output pair.
Conclusion: Tom: So looking at the whole paper "Understanding Sample Efficiency in Predictive Coding," it seems they’re providing a solid mathematical foundation for why predictive coding is superior to backpropagation in terms of sample efficiency.
Jane: The implication for us is that if we use this layer-specific learning rate approach, we can build models that learn more effectively from fewer data points and without getting bogged down by weight interference.
Lu: It suggests a path forward for designing architectures where the learning process itself is guided by these alignment properties rather than just minimizing a standard squared error energy function.
Meng: From an engineering standpoint, it means we can design training schedules that are more robust, potentially reducing the amount of data you need to get good performance out of a network on real traffic.
Lalam: The paper gives us concrete conditions—those layer-specific learning rates—that guarantee this interference-free learning for single samples and batch learning, which is a very practical result.
Tom: It’s not saying predictive coding is the only way to do things, but it does provide a strong mechanistic understanding of why it works better in these specific contexts.
Jane: The authors give us tools to guide how we should parameterize predictive coding for the best learning outcomes, which is really helpful for anyone trying to apply these ideas.
More episodes
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation
- 2606.10592-Dirichlet-Guided Group Forecasting for Alleviating Over-smoothing in Time Series Forecasting
- 2606.11480-FedRAN: Accurate and Efficient Analytic Federated Continual Learning
- 2606.15160-DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- 2606.20937-Learning through Internalization
- 2606.24152-Autonomous Video Generation for Self-Evolving World Models
- 2606.26432-Embedding Foundation Model Predictions in Discrete-Choice Models with Structural Guarantees