It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
That's the same impression I had. I was afraid I am not getting something or missing a bigger picture. I am glad I am not the only one who feels that way.
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
This command is a lot of words to say "I don't like it" without giving any reason to believe you. If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…
Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.
Not a deep learning expert, but: it seems that without backpropagation for model updates, the communication costs should be lower. And that will enable models that are easier to parallelize? Nvidia isn't creating new versions of its NVLink/NVSwitch products just for the sake of it, better communication must be a key enabler. Can someone with deeper knowledge can comment on this? Is communication a bottleneck, and wil…
> will this algorithm uncover a new design space for NNs? No. Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse. The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them. Each computational unit is independen…
Yeah, no. Reading the paper I don't really see anything but a superficial resemblance to stacking. Hinton was active back when Wolpert introduced stacking and I'm fairly sure he is aware of it. If anything it much more closely resembles his own prior work in Boltzmann machines, unsurprisingly (and which he cites), or even his prior work on capsules. I don't know if this will really pan out into anything that different or useful for the field, but it's unfair and inaccurate to dismiss it as derivative of stacking.
It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
Since you seem to understand what he is saying, can you explain to me how the per layer objective function looks like? I don't get what he means by inserting the label into the input and what labels he is using per layer.
A model is a function F that minimizes error in y_i = F(X_i) + error. Inserting a label simply means a function F(X_i,y_j). Then you optimize it in some way to separate true labels from false label, e.g. F(X_i, y_j) = (y_i==y_j)-(y_i!=y_j) + error.
I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…
This command is a lot of words to say "I don't like it" without giving any reason to believe you. If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
We tend to start hallucinating when we don't have enough sleep. So generating training data is necessary, but way safer when our muscles are turned off.