51–60 of 130 posts
Earlier quoted context omitted.
Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.
Just curious, why would one want to avoid citing Schmidhuber's work?
Earlier quoted context omitted.
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
Yeah, no. Reading the paper I don't really see anything but a superficial resemblance to stacking. Hinton was active back when Wolpert introduced stacking and I'm fairly sure he is aware of it. If anything it much more closely resembles his own prior work in Boltzmann machines, unsurprisingly (and which he cites), or even his prior work on capsules. I don't know if this will really pan out into anything that differen…
Since gradients don't flow from B to A in (B.f.A)(x), A is trained independently of B, meaning that the training distribution of B changes without B influencing it, i.e. context drift. B doesn't know the difference, and B doesn't influence it.
For all intents and purposes, you can compute all the outputs as training of A happens, meaning training A to completion, and then feed them into B and B will still compute the same outputs and derivatives as it did before.
To deal with context drift, Hinton proposes normalizing the data, so the distribution does not change significantly.
Whatever he proposed is not "backprop-free" either. It still involves backprop, but the number of layers gradients flow through is 1, the layer itself.
The argument that you can still train through non-differentiable operations is not particularly convincing either; the reparameterization trick shows that is trivial to pass gradients through non differentiable operations if we are smart about it.
Given non differentiable operator Z: R^N -> R^N; let A, B, C be R^N -> R^N linear layer, B(Z(C(x)) * A(C(x))) allows gradients to flow through B and A all the way to C. The output of Z is for all intents and purposes a Hadamard product with (A . C)(x) that is runtime constructed and might as well be part of the input.
You can even run Z(C(x)) through a neural network and learn how to transform that and still provide useful and informative gradients back to C(x) via (A . C)
Earlier quoted context omitted.
Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of…
> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negativ…
The implementations of this compute gradients locally.
It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
Discussion last month when the preprint was released: https://news.ycombinator.com/item?id=33823170
Deep dive tutorial for learning in a forward pass [1] [1] https://amassivek.github.io/sigprop
I don't get it, don't all of those optimizers work via backprop?
Earlier quoted context omitted.
This command is a lot of words to say "I don't like it" without giving any reason to believe you. If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
It performs worse than baсkprop.
Earlier quoted context omitted.
> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negativ…
Let's suppose that you are correct, which direction are the weights updated towards? The implementations of this compute gradients locally.
Earlier quoted context omitted.
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
[flagged]
Earlier quoted context omitted.
It performs worse than baсkprop.
Is it slower and less accurate ? Or just slower.