Geoffrey Hinton publishes new deep learning algorithm
1–10 of 130 posts
Re: Geoffrey Hinton publishes new deep learning algorithm
#2Direct link to an implementation on GitHub: https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler...
--
The divulgational title is almost an understatement: the Forward-Forward algorithm is an alternative to backpropagation.
Edit: sorry, the previous formulation of the above in this post, relative to the advantages, was due to a misreading. Hinton writes:
> The Forward-Forward algorithm (FF) is comparable in speed to backpropagation but has the advantage that it can be used when the precise details of the forward computation are unknown. It also has the advantage that it can learn while pipelining sequential data through a neural network without ever storing the neural activities or stopping to propagate error derivatives....The two areas in which the forward-forward algorithm may be superior to backpropagation are as a model of learning in cortex and as a way of making use of very low-power analog hardware without resorting to reinforcement learning
Re: Geoffrey Hinton publishes new deep learning algorithm
#3Direct link to an implementation on GitHub: https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler... -- The divulgational title is almost an understatement: the Forward-Forward algorithm is an alternative to backpropagation. Edit: sorry, the previous formulation of the above in this post, relative to the advantages, was due to a misreading. Hinton writes: > The Forward-Forward algorithm (FF) is comparable in s…
[deleted]
Re: Geoffrey Hinton publishes new deep learning algorithm
#4This is an interesting approach and I have read that this is more closer to how our brains works.
We extract learning, while we are imbibing the data and there seems to be no mechanism in the brain that favors backprop like learning process.
Re: Geoffrey Hinton publishes new deep learning algorithm
#5Re: Geoffrey Hinton publishes new deep learning algorithm
#6Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize?
Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler.
Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and will this algorithm uncover a new design space for NNs?
Re: Geoffrey Hinton publishes new deep learning algorithm
#7Deep dive tutorial for learning in a forward pass [1]
Re: Geoffrey Hinton publishes new deep learning algorithm
#8Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
It’s more that without backpropagation, you no longer need to store your forward activations across many layers to compute the backwards pass which usually is dependent on a forward pass. When a network is hundreds of layers, and batches are very large, the forwards and backwards accumulations add up in terms of memory required.
Communication across GPUs doesn’t solve this but instead allows to have either many models running in parallel on different GPUs to increase Barch size or to share many layers across GPUs to increase model size. Quick communication is critical to maintain training speeds that aren’t astronomical
Re: Geoffrey Hinton publishes new deep learning algorithm
#9It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor).
As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift.
To deal with the context drift, Hinton proposes to normalise the output.
This isn't anything new or novel. Expressing "ThIs LoOkS sImIlAr To HoW cOgNiTiOn WoRkS" to make it sound impressive doesn't make it impressive or good by any stretch of the imagination.
Hinton just took something that existed for a long time, made it worse, gave it a different name and wrapped it in a paper under his name.
With every paper I am more convinced that the Laureates don't deserve the award.
Sorry, this "paper" smells from a mile away, and the fact that it is upvoted as much shows that people will upvote anything if they see a pretty name attached.
Edit:
Due to the apparent controversy of my criticism, I can't respond with a reply, so here is my response to the comment below asking what exactly makes this worse.
> As far as I can tell, this is almost the same as stacking multiple layers of ensembles
It isn't new. Ensembling is used and has been used for a long time. All kaggle competitions are won through ensembles and even ensembles of ensembles. It is a well studied field.
> except worse as each ensemble is trained while previous ensembles are learning.
Ensembles exhibit certain properties, but only iff they are trained independently from each other. This is well studied, you can read more about it in Bishop's Pattern recognition book.
> This is causing context drift.
Context drift occurs when a distribution changes over time. This changes the loss landscape which means the global minima change / move.
> To deal with the context drift, Hinton proposes to normalise the output.
So not only is what Hinton built a variation of something that existed already, made it worse by training the models simultaneously, and to handle the fact that it is worse, he adds additional computations to deal with said issue.
Re: Geoffrey Hinton publishes new deep learning algorithm
#10Isn't this similar to how GAN networks learn? Edit: Yes, there is small chapter in paper comparing it to GAN