21–30 of 130 posts
This is an interesting approach and I have read that this is more closer to how our brains works. We extract learning, while we are imbibing the data and there seems to be no mechanism in the brain that favors backprop like learning process.
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
Earlier quoted context omitted.
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
You're accusing one of the foundations of modern AI with either being a fraud, or incompetent. At best that seems short sighted, no?
Earlier quoted context omitted.
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
[flagged]
So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent?
So then what is the label for each layer? Do you use the same label for each layer?
And what does he mean by the forward pass not being fully known? I don't get this application of the blackbox between layers. Why would you want to do that?
Earlier quoted context omitted.
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
You're accusing one of the foundations of modern AI with either being a fraud, or incompetent. At best that seems short sighted, no?
I could name many other people who have actually been more influential in the field.
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
I don't get what he means by inserting the label into the input and what labels he is using per layer.