Live data from Hacker News

Predictive coding has been unified with backpropagation

lesswrong.com

51–60 of 85 posts

Re: Predictive coding has been unified with backpropagation

#51
post #13

EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…

Interesting follow up reading:

"Relaxing the Constraints on Predictive Coding Models" (https://arxiv.org/abs/2010.01047), from the same authors. Looks at ways to remove neurological implausibility from PCM and achieve comparable results. Sadly they only do MNIST in this one, and are not as ambitious in testing on multiple architectures and problems/datasets, but the results are still very interesting and it covers some of the important theoretical and biological concerns.

"Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks" (https://arxiv.org/abs/2103.03725), from different authors. Uses an alternative formulation that means it always converges to the backprop result within a fixed number of iterations, rather than approximately converges "in practice" within 100-200 iterations. Not only is this a stronger guarantee, it means they achieve inference speeds within spitting distance of backprop, levelling the playing field. (Edit: also noted by eutropia)

It'd be interesting to see what a combination of these two could do, and at this point I feel like a logical next step would be to provide some setting in popular ML libraries such that backprop can be switched for PCM. Being able to verify this research just be adding a single extra line for the PCM version, and perhaps replicating state-of-the-art architectures, would be quite valuable.

Re: Predictive coding has been unified with backpropagation

#53
post #49

Earlier quoted context omitted.

Indeed, it's worth mentioning we still have absolutely no idea how memory works.

we know a lot about memory, but most AI researchers are simply ignorant in neuroscience or cognitive psychology and stick with their comfort zone. Saying "we have no idea" is just being lazy.

No. We really have no idea what is going on. We only know some basic psychology about it (holding 7 things in short term, etc.) If we knew something about implementation, we could implement human-like memory.

Re: Predictive coding has been unified with backpropagation

#54
post #25
post #13

EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…

the thing is about every week there is a paper published with groundbreaking claims, with this question in particular being very popular, trying to unify neuroscience and deep learning in some way, in search for computational foundations of AI. Mostly this is driven by success of DL in certain industrial applications. Unfortunately most of these papers are heavy on theory but light on empirical evidence. If we follow…

Not only light on evidence, but essential practicality-free. There's no "there" there. Literally nothing useful will come from this.

Re: Predictive coding has been unified with backpropagation

#55
post #37

Earlier quoted context omitted.

The article says this: > The backpropagation algorithm requires information to flow forward and backward along the network. But biological neurons are one-directional. An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. An axon potential never travels backward from a cell's terminals to its body. The point of the research here is that backpropagation turns out…

So... I don't understand. >An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. How do you figure that doesn't allow backprop? A neuronal bit is a loop of neurons. Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Neuron fires dendrite to axon, secondary neuron fires…

> How do you figure that doesn't allow backprop?

When you backprop through a linear layer (a matrix W), you need to multiply with W.transpose which is impossible if connections are not bidirectional.

> Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything?

Local error aggregation can have a similar effect with backprop but you can run each layer in parallel, you don't need to wait for the signal to reach the loss function and then gradients to come all the way back.

An interesting read: Decoupled Neural Interfaces using Synthetic Gradients (DeepMind, 2017) http://proceedings.mlr.press/v70/jaderberg17a/jaderberg17a.p...

Re: Predictive coding has been unified with backpropagation

#56
post #44

Earlier quoted context omitted.

Hurry and publish before I have manuscripts ready applying these results.

Hey, Eli :-) I'm working on it; I'll send you an e-mail. Things quickly turned out to be more general than I realized last year.

That would make sense. The whole ACT/categorical cybernetics community has been working out how massively general optics are :-).

Re: Predictive coding has been unified with backpropagation

#57
post #49

Earlier quoted context omitted.

we know a lot about memory, but most AI researchers are simply ignorant in neuroscience or cognitive psychology and stick with their comfort zone. Saying "we have no idea" is just being lazy.

No. We really have no idea what is going on. We only know some basic psychology about it (holding 7 things in short term, etc.) If we knew something about implementation, we could implement human-like memory.

I suggest starting with the works by Howard Eichenbaum on memory and Edvard & May-Britt Moser (and John O'Keefe and Lynn Nadel) on place & grid cells.

For the latest and greatest see

https://twitter.com/doellerlab

https://twitter.com/KordingLab

https://twitter.com/preston_lab

https://twitter.com/memorylab

https://twitter.com/ptoncompmemlab

https://twitter.com/MillerLabMIT

https://twitter.com/hugospiers

Once you start pulling that thread you'd be surprised how much we do know.

Re: Predictive coding has been unified with backpropagation

#58
post #13

EDIT: Before you read my comment below, please see https://news.ycombinator.com/item?id=26702815 and https://openreview.net/forum?id=PdauS7wZBfC for a different view. -- If the results hold, they seem significant enough to me that I'd go as far as saying the authors of the paper would end up getting an important award at some point, not just for unifying the fields of biological and artificial intelligence , but also…

Note that the paper was rejected for publication in ICLR 2021: https://openreview.net/forum?id=PdauS7wZBfC

That is an awesome site, thanks for posting I had no idea there was a place with that much transparent review (shows how much I've been publishing).

Re: Predictive coding has been unified with backpropagation

#59
post #37

Earlier quoted context omitted.

The article says this: > The backpropagation algorithm requires information to flow forward and backward along the network. But biological neurons are one-directional. An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. An axon potential never travels backward from a cell's terminals to its body. The point of the research here is that backpropagation turns out…

So... I don't understand. >An action potential goes from the cell body down the axon to the axon terminals to another cell's dendrites. How do you figure that doesn't allow backprop? A neuronal bit is a loop of neurons. Information absolutely can back- propagate. If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything? Neuron fires dendrite to axon, secondary neuron fires…

> If it couldn't, how does anyone think it'd be at all possible to learn how to get better at anything?

Presumably part of the feedback loop (at least for things like motor skills and rote memorisation) is external to the brain. Our brain causes us to act, which alters our perceptions, which causes the brain to adjust.

Post reply on HN