41–50 of 130 posts
This is an interesting approach and I have read that this is more closer to how our brains works. We extract learning, while we are imbibing the data and there seems to be no mechanism in the brain that favors backprop like learning process.
Fact: Geoffrey Hinton has discovered how the brain works. Every few years actually.
I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…
Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.
What exactly is the negative data? Seems like it's just scrambled nonsense (aka what the truth is not)
I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…
The idea is imo similar to how random word embeddings are generated.
Quantum would absolutely change everything in DL/ML space.
It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
We tend to start hallucinating when we don't have enough sleep. So generating training data is necessary, but way safer when our muscles are turned off.
This makes me cheerful because it suggests a way that studying systems which appear intelligent might be able to teach us more about how human intelligence works.
If these FF networks can be proven to scale or made to scale similarly to BP networks, this would enable making hardware several orders of magnitude more efficient, for the price of loosing the ability to make exact copies of models to other computers. (The loss of reproducibility sits well with the tradition of scientific papers anyway/s;)
2.) How does this paper relate to Hintons feedback alignment from 5 years ago? I remember it was feedback without derivatives. What are the key new ideas? To adjust the output of each individual layer to be big for positive cases and small for negative cases without any feedback? Have these approaches been combined?
I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…
Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.
I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…
Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of…
I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely:
The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself
It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
We tend to start hallucinating when we don't have enough sleep. So generating training data is necessary, but way safer when our muscles are turned off.