Predictive coding has been unified with backpropagation
1–10 of 85 posts
Re: Predictive coding has been unified with backpropagation
#2This is an overly strong claim for the paper (which is good!) backing it.
If anyone is interested in the reader's digest version of the original paper check out https://www.youtube.com/watch?v=LB4B5FYvtdI
Re: Predictive coding has been unified with backpropagation
#3I'm glad people are talking about this, and the similarity between predictive coding and the action of biological neurons is interesting. But we shouldn't fetishize predictive coding. There's a wider discussion going on, and several theories as to how back propagation might work in the brain.
https://www.cell.com/trends/cognitive-sciences/fulltext/S136...
Re: Predictive coding has been unified with backpropagation
#4At scale, Evolutionary Strategies (ES) are a very good approximation of the gradient as well. Don’t recommend to jump just yet to conclusions and unifications.
Re: Predictive coding has been unified with backpropagation
#5Interesting discussion on the ICLR openreview, resulting in a reject:
Re: Predictive coding has been unified with backpropagation
#6At scale, Evolutionary Strategies (ES) are a very good approximation of the gradient as well. Don’t recommend to jump just yet to conclusions and unifications.
The author's point is that predictive coding is a plausible mechanism by which biological neurons work. ES are not.
ANNs have deviated widely from their biological inspiration, most notably in the way that information flows, since backpropagation requires two way flow and biological axons are one-directional.
If predictive coding and backpropagation are shown to have similar power, then there's a rough idea that the way that ANNs work isn't too far from how brains work (with lots and lots of caveats).
Re: Predictive coding has been unified with backpropagation
#7I'm glad people are talking about this, and the similarity between predictive coding and the action of biological neurons is interesting. But we shouldn't fetishize predictive coding. There's a wider discussion going on, and several theories as to how back propagation might work in the brain. https://www.cell.com/trends/cognitive-sciences/fulltext/S136... https://www.nature.com/articles/s41583-020-0277-3
there is no evidence of back-propagation in the brain.
See Professor Edmund T. Rolls books on biologically plausible neural networks:
"Brain Computations: What and How" (2020) https://www.amazon.com/gp/product/0198871104
"Cerebral Cortex: Principles of Operation" (2018) https://www.oxcns.org/b12text.html
"Neural Networks and Brain Function" (1997) https://www.oxcns.org/b3_text.html
Re: Predictive coding has been unified with backpropagation
#8Yeah, I don't like this title. Coding for backprop is worth getting excited about, but please don't assume it supersedes all forms of "predictive coding". Plenty of predictive learning techniques do just fine without it, including our own brains.
In keeping with the No-Free-Lunch theorem, it's also highly desirable in general to have a variety of approaches at hand for solving certain predictive coding problems. Yes, this makes ML (as a field) cumbersome, but it also prevents us from painting ourselves into a corner.
Re: Predictive coding has been unified with backpropagation
#9At scale, Evolutionary Strategies (ES) are a very good approximation of the gradient as well. Don’t recommend to jump just yet to conclusions and unifications.
The author's point is that predictive coding is a plausible mechanism by which biological neurons work. ES are not. ANNs have deviated widely from their biological inspiration, most notably in the way that information flows, since backpropagation requires two way flow and biological axons are one-directional. If predictive coding and backpropagation are shown to have similar power, then there's a rough idea that the…
> If predictive coding and backpropagation are shown to have similar power, then there's a rough idea that the way that ANNs work isn't too far from how brains work (with lots and lots of caveats).
So many caveats that I don't even really think that is a true statement.
Re: Predictive coding has been unified with backpropagation
#10I'm glad people are talking about this, and the similarity between predictive coding and the action of biological neurons is interesting. But we shouldn't fetishize predictive coding. There's a wider discussion going on, and several theories as to how back propagation might work in the brain. https://www.cell.com/trends/cognitive-sciences/fulltext/S136... https://www.nature.com/articles/s41583-020-0277-3
there is no evidence of back-propagation in the brain. See Professor Edmund T. Rolls books on biologically plausible neural networks: "Brain Computations: What and How" (2020) https://www.amazon.com/gp/product/0198871104 "Cerebral Cortex: Principles of Operation" (2018) https://www.oxcns.org/b12text.html "Neural Networks and Brain Function" (1997) https://www.oxcns.org/b3_text.html
"There is just one problem: [biological neural networks] are physically incapable of running the backpropagation algorithm."
From the linked article.