61–70 of 130 posts
This is an interesting approach and I have read that this is more closer to how our brains works. We extract learning, while we are imbibing the data and there seems to be no mechanism in the brain that favors backprop like learning process.
Fact: Geoffrey Hinton has discovered how the brain works. Every few years actually.
Earlier quoted context omitted.
Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of…
> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negativ…
Earlier quoted context omitted.
Let's suppose that you are correct, which direction are the weights updated towards? The implementations of this compute gradients locally.
It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?
In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters.
This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.
Earlier quoted context omitted.
It performs worse than baсkprop.
So the initial “hey, this might be a good idea” implementation performs slightly worse than something that has had literally billions of dollars thrown at it?
No comparisons to AdamW were made.
In fact, this algorithm uses backprop at its core, but propagating through 0 layers.
Earlier quoted context omitted.
It performs worse than baсkprop.
So the initial “hey, this might be a good idea” implementation performs slightly worse than something that has had literally billions of dollars thrown at it?
Earlier quoted context omitted.
Just curious, why would one want to avoid citing Schmidhuber's work?
It is a bit of a meme in AI research, as Schmidhuber often claims that he hasn't received the citations that he thinks he deserves. https://www.urbandictionary.com/define.php?term=schmidhuber
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial.
So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using another optimization approach and achieving similar accuracy?
It's a unique and interesting approach, but the article specifically mentions it gets accuracy similar to backprop, but if this is the experiment that claim is based on, it loses some credibility in my eyes.