Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks

The gist The pairing of bidirectional PC (bPC) and error-based reparameterization (ePC) enables deep, 4-bit predictive coding networks that retain all canonical properties at depths up to eight layers.

Background

Predictive coding is a hierarchical learning algorithm rooted in the free-energy principle where each layer minimizes the discrepancy between its own activity and a top-down prediction from the layer above. It possesses three properties that backpropagation does not offer: local learning, native generation, and flexible, energy-based inference. Canonical state-based PC (sPC) suffers from an exponential decay of the inference signal with depth, which causes deep models to converge slowly or not at all. Aggressive quantization, such as four bits, is the de facto precision for efficient inference and training across large model families.

Method

The proposed method combines bidirectional PC (bPC) and error-based reparameterization (ePC), referred to as ePC+bPC, to achieve the PCN property set in the deep, 4-bit regime. The core idea is to optimize prediction errors rather than neural states during inference.

The network structure involves top-down (generative) weights W and bottom-up (discriminative) weights V. In training, the inner loop adjusts stored per-layer prediction errors instead of states, which allows a single global backpropagation pass to deliver the output-loss signal to all layers at once. The outer learning step then updates the weights using Adam on these fixed errors.

Key Findings and Mechanisms

The ePC+bPC model successfully retains the PCN property set at depths up to at least eight layers with W4A4E4 QAT. This is the regime where canonical PC fails outright, and it maintains full-precision accuracy together with clean class-conditional generation under W4A4E4 QAT across FP4 and three posit formats.

The method overcomes two distinct failure modes: bounded signal decay in state-based PC, where the network stalls because the inference signal shrinks geometrically, and error divergence in unregularized error optimization. Error optimization delivers the output signal to every layer in a single backward pass, removing the depth-wise signal decay. To stabilize the error dynamics at depth, global-norm clipping of the error gradients is required.

Quantization and Precision Analysis

The study characterizes four 4-bit formats (FP4 E2M1, Posit⟨4,0⟩, Posit⟨4,1⟩, Posit⟨4,2⟩). The analysis shows that 4-bit quantization is nearly free for ePC+bPC: full-precision and 4-bit accuracies differ by at most ∼5 points. The worst format is Posit⟨4,2⟩, which opens the widest gap under the bulk of the data, jumping eightfold from 0.125 to 1. The usable conclusion for a hardware designer is that any of these formats will serve as a floor rather than an ordering.

Implications for Hardware Co-design

The results suggest that a 4-bit-only datapath is viable for deep PC, with W4A4E4 ePC+bPC matching its FP32 self to within ∼5% at six layers. Furthermore, the method adds a backward pass per inference step, which trades away strict neuromorphic locality but allows the output signal to reach every layer at once. This is considered the right trade for a digital datapath.

Conclusion

The appeal of predictive coding has always been a set of properties that matter most on exactly the deep, low-precision regime where canonical PC fails. The combination of bidirectional PC with error-based reparameterization and gradient clipping yields networks that train to at least eight layers and retain their full-precision accuracy with clean generation under W4A4E4 QAT. This result clears a practical path to a deep, 4-bit-native datapath for PCN accelerators. The obstacles are inference-signal decay and error-optimization stability, not the often-assumed quantization of errors to zero.

How it works

The network is a stack of layers carrying two sets of weights between each neighbouring pair, top-down (generative) weights W and bottom-up (discriminative) weights V. In training, the inner loop adjusts the stored per-layer prediction errors instead of states to lower the bidirectional energy. This involves recovering layer states from errors by a single generative feed-forward pass and then backpropagating the resulting energy gradient to every stored error in one backward pass.

Failure Modes

The two methods fail for opposite reasons. State-based bPC fails by bounded decay, where converged error magnitudes stay O(1) at every depth but the inference signal shrinks geometrically, causing the network to stall. Unregularized ePC+bPC fails by divergence because errors are directly optimized and states depend on them without bound, causing the converged error magnitude to explode with depth. Global-norm clipping of the error gradients removes this divergence, holding the error magnitude at ∼10−4 across all depths.

Ablation Results

Removing the error reparameterization collapses accuracy and generation, while removing bidirectionality recovers high accuracy but forfeits generation. Removing gradient clipping from ePC+bPC reintroduces the divergence of §4.4. Only the full method is both accurate and generative at depth and 4 bits.

Limitations

Absolute accuracies are further held down by the one-epoch training budget. Beyond L≈6, ePC+bPC accuracy declines, indicating that the stabilizing effect of a single global-norm clipping constraint weakens at extreme depth. The experiments are also CPU-bound, which capped the practical sweep scale. This underscores the work’s motivation: a native low-precision PCN accelerator is precisely what would make deep PCNs practical at scale.

Conclusion

The appeal of predictive coding has always been a set of properties that matter most on exactly the deep, low-precision regime where canonical PC fails. We have shown that this property set survives into that regime by pairing bidirectional PC with the error-based reparameterization, and stabilizing the error dynamics by clipping the norm of the error gradients. Combined, these methods yield networks that train to at least eight layers and retain their full-precision accuracy with clean generation under W4A4E4 QAT. We further clarified the mechanism: the obstacles are inference-signal decay and error-optimization stability, not the often-assumed quantization of errors to zero. The result clears a practical path to a deep, 4-bit-native datapath for PCN accelerators.

References

[1] Karl Friston. A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences, 360(1456):815–836, 2005 <ref:

1. Bidirectional inference in a single network allows for unified discrimination and generation, enabling one circuit that both discriminates, generates, denoises, and completes missing data. This means the system can simultaneously classify an input (discriminative pathway) while producing a coherent output (generative pathway).

2. Error-based reparameterization eliminates the depth-wise signal decay in inference by optimizing prediction errors rather than states, which allows for a single backward pass to deliver the output-loss signal to every layer in a single backward pass. This removes the barrier of deep networks failing due to vanishing learning signals.

3. Global-norm clipping of error gradients stabilizes the system, as it addresses the divergence issue by bounding error explosions: Global-norm clipping of the error gradients removes the divergence at all depths and formats (Fig. 4). This ensures stable training even when optimizing errors in deep models up to eight layers.

4. The W4A4E4 quantization-aware training (QAT) scheme enables high accuracy with low bit-width: full-precision and 4-bit accuracies differ by at most ∼5 points, and by under two in the best-performing formats. This allows for deploying deep predictive coding networks on hardware with minimal loss of performance.

5. The resulting system can train to at least eight layers while retaining full-precision accuracy together with clean class-conditional generation under W4A4E4 QAT, enabling the creation of highly accurate, generative models in a resource-constrained environment.

Predictive coding networks (PCNs) promise properties not offered by backpropagation-trained networks: learning from purely local information, a single network that both discriminates and generates, and flexible inference by energy minimization. However, their iterative inference is expensive, making hardware acceleration desirable, and a useful, efficient accelerator needs both depth and aggressive quantization. Canonical state-based PC delivers neither: its inference signal decays exponentially with depth, so deep models fail to train, and low precision makes it worse. We show that pairing two recent methods carries the full PCN property set into the deep, 4-bit regime: bidirectional PC (bPC), which unifies discrimination and generation, and the error-based reparameterization (ePC), which optimizes prediction errors rather than states during inference. Error optimization delivers the output signal to every layer in a single backward pass, removing the signal decay; one additional element, global-norm clipping of the error gradients, stabilizes the error dynamics at depth. The resulting ePC+bPC model trains stably up to at least eight layers and retains its full-precision accuracy and clean class-conditional generation under W4A4E4 quantization-aware training, across FP4 and three posit formats. On six-layer networks, where state-based bPC collapses to chance, ePC+bPC classifies more than 50 accuracy points above it after one epoch of training, with class-conditional generations 5-9x better in RMSE. 4-bit quantization is nearly free: full-precision and 4-bit accuracies differ by at most 5 points, and by under two in the best-performing formats. We characterize the two failure modes the method overcomes, bounded signal decay in state-based PC and error divergence in unregularized error optimization, and discuss implications for a 4-bit-native PCN accelerator.