Exploring the Inner Workings and Internal Representations of Predictive Coding Networks in Comparison to Usual Feedforward Neural Networks

Authors: Aslan Satary Dizaji

Presenting author: Aslan Satary Dizaji

Presentation type: Poster at SNUFA 2025 online workshop (5-6 Nov 2025)

Abstract

Predictive coding theory as a unified theory of brain formulating intelligence as a hierarchical process in which the brain builds an unsupervised world model and tries to predict the next states of the world by minimising the prediction errors. This process can be implemented in a variant of artificial neural networks, called energy-based networks or predictive coding networks, trained by a biologically plausible learning rule named prospective configuration. In each step of prospective configuration, first the neuronal activities of intermediate layers are adjusted to reflect the activities required to produce the targets and then synaptic weights of intermediate layers are adjusted to consolidate these neuronal activities. While in recent years, there is a good progress on theoretical understanding of predictive coding networks trained by prospective configuration algorithm, the inner machinery and the internal representations of these neural networks compared to usual feedforward neural networks are still unknown. This project aims to fill this gap using interpretability techniques for vision tasks. Basically, a few recently published methods are used to measure the internal representations of predictive coding networks to compare them with usual feedforward neural networks. The project is performed by simple vision tasks such as relatively simple predictive coding networks trained on relatively simple synthesised or well-known datasets. The interpretability methods or techniques that are used include measuring disentanglement and dimensionality of internal representations across layers, convexity analysis of intermediate dense layers, measuring factorisation and invariance of internal representations across layers, measuring modularity and flexibility of internal representations during training, representational similarity analysis across layers, and analysis and mechanism of feature learning using coding schemes and deep neural feature ansatz respectively. Additionally, I will quantify if predictive coding networks using a local learning rule such as prospective configuration compared to usual feedforward neural networks is able to bypass the curse of dimensionality of sparse distributed data. Moreover, I will measure the tradeoff between disentanglement and entanglement of predictive coding networks and usual feedforward neural networks during training for better generalisation. Predictive coding theory and its instantiation in artificial neural networks via prospective configuration algorithm is a top-down theory of brain theorising how the brain must work. On the other hand, there is a recent bottom-up theory of brain using the neuronal least-action principle to posit how predictive coding algorithm is actually instantiated in the brain. Comparing the internal representations of these top-down and bottom-up theories of brain is another major goal of this project in future. This project is an exploratory project without having any concrete hypothesis in advance regarding the internal representations in predictive coding networks in comparison to usual feedforward neural networks, while the authors generally believe that the internal representations of these neural networks are probably more brain-like. Also, at the moment, the project is in early steps and there are some non-conclusive preliminary results which they will be further expanded in the coming weeks and months.