11–20 of 130 posts
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
[deleted]
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
No.
Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse.
The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them.
Each computational unit is independent, meaning that you don't calculate gradients going outside of it.
This is the same as training a bunch of networks, computing predictions, and then using another layer to combine the predictions. This is called stacking, and the networks are called an "ensemble". You can do this multiple times and have N levels of meta estimators.
Instead of fitting the ensemble and then the meta estimator, Hinton proposes training both simultaneously but without allowing gradients to flow through.
That is stupid because if you don't allow gradients to flow through, you will see a context drift as the data distribution changes. Hinton observed this context drift, to deal with that, he proposed normalizing the data.
On one extreme, you can use individual linear units as the models, and on the other extreme, you can combine all units into a single neural network and treat that as a module.
So no, this does not open any new design, it's an old idea, worsened, and wrapped in fancy words and post-facto reasoning.
If you are curious how a linear layer is an ensemble, observe that each vector is its own linear estimator, making the linear mapping an ensemble of estimators.
Isn't this similar to how GAN networks learn? Edit: Yes, there is small chapter in paper comparing it to GAN
Quantum would absolutely change everything in DL/ML space.
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
"Context drift as data distribution changes" sounds a hell of a lot like real life to me.
Normalized = hedonic treadmill on long view
At the micro scale, data that overflows the normalization is stored in emotional state, creating an orthogonal source of truth that makes up for the lack of full connected learning.
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
I remember a long time ago there was a collection of Geoffrey Hinton facts. Like "Geoffrey Hinton once wrote a neural network that beat Chuck Norris."