Interesting discussion on the ICLR openreview, resulting in a reject: https://openreview.net/forum?id=PdauS7wZBfC
The review is great, it contains all the interesting points and counterpoints, in a much more succinct format than the article itself.
Predictive coding has been unified with backpropagation
21–30 of 85 posts
Re: Predictive coding has been unified with backpropagation
#22EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…
Here's a more recent paper (March, 2021) which cites the above paper: https://arxiv.org/abs/2103.04689 "Predictive Coding Can Do Exact Backpropagation on Any Neural Network"
Yup. I'd expect to see many more citations going forward. In particular, I'd be excited to see how this ends up getting used in practice, e.g., training and running very large models running on distributed, masively parallel "neuromorphic" hardware.
Re: Predictive coding has been unified with backpropagation
#23Interesting discussion on the ICLR openreview, resulting in a reject: https://openreview.net/forum?id=PdauS7wZBfC
Another well received paper [1], but I want to point out that ICLR should really have an industry track.
The type of research in [1] (exhaustive analytic study on various parameters on RL training), is clearly beyond typical academia environment, probably also beyond normal industry labs. Note the paper was from Google Brain.
The study consumes a lot of people's time, and computing time. It's no doubt very useful and valuable. But I dont think they should be judged by the same group of reviewers with the other work from normal universities.
Re: Predictive coding has been unified with backpropagation
#24I think that this sort of forward backward thing is a very general idea. There’s a one to many relationship called the adjoint, and a many to one relationship called the norm.
I wrote something about this here https://github.com/adamnemecek/adjoint
Re: Predictive coding has been unified with backpropagation
#25EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…
the thing is about every week there is a paper published with groundbreaking claims, with this question in particular being very popular, trying to unify neuroscience and deep learning in some way, in search for computational foundations of AI. Mostly this is driven by success of DL in certain industrial applications.
Unfortunately most of these papers are heavy on theory but light on empirical evidence. If we follow the path of natural sciences, theory has to agree with evidence. Otherwise it's just another theory unconstrained by reality, or worse, pseudo-science.
Re: Predictive coding has been unified with backpropagation
#26EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…
I’m going to personally flog any researcher who titles their next paper “Predictive Coding Is All You Need”. You’ve been warned.
Re: Predictive coding has been unified with backpropagation
#27> Predictive coding is the idea that BNNs generate a mental model of their environment and then transmit only the information that deviates from this model. Predictive coding considers error and surprise to be the same thing. Hebbian theory is specific mathematical formulation of predictive coding.
This is an excellent, concise explanation. It sounds intuitive as something that could work. Would love to try and dabble with this. Any resources?
Re: Predictive coding has been unified with backpropagation
#28EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…
I’m going to personally flog any researcher who titles their next paper “Predictive Coding Is All You Need”. You’ve been warned.
There are already 60+ of those, and counting, all but one of them since Vaswani et al's transformer paper:
https://arxiv.org/search/?query=is+all+you+need&searchtype=a...
Re: Predictive coding has been unified with backpropagation
#29Does anyone know of a simple code example that demonstrates the original predictive coding concept from 1999? Ideally applied to some type of simple image/video problem.
I thought I saw a Matlab explanation of that 99 paper but have not found it again.
Re: Predictive coding has been unified with backpropagation
#30EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…
the thing is about every week there is a paper published with groundbreaking claims, with this question in particular being very popular, trying to unify neuroscience and deep learning in some way, in search for computational foundations of AI. Mostly this is driven by success of DL in certain industrial applications. Unfortunately most of these papers are heavy on theory but light on empirical evidence. If we follow…
The paper (arxiv:2103.04689) linked by eutropia above has some empirical evidence on the ML side, showing that performance of predictive coding is not so far off backprop. And there is no shortage of suggestions for how neural circuits might work around the strict requirements of backprop-like algorithms.
cs702's original comment above is excessively hyperbolic: the compositional structure of Bayesian inversion is well known and is known to coincide structurally with the backward/forward structure of automatic differentiation. And there have been many papers before this one showing how predictive coding approximates backprop in other cases, so it is no surprise that it can do so on graphs, too. I agree with the ICLR reviewers that this paper is borderline and not in itself a major contribution. But that does not mean that this whole endeavour, of trying to find explicit mathematical connections between biological and artificial learning, is ill motivated.