71–80 of 130 posts
It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.
Most of the "training process" of our brain likely occurred prior to our birth in evolutionarily optimized structure of brain.
Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…
Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…
You have to start with toy models before scaling up.
Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.
It seems that the point is that the objective function is applied layerwise, still computes gradient to get the update direction, it's just that gradients don't propagate to previous layers (detatched tensor). As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift. To deal wit…
Since you seem to understand what he is saying, can you explain to me how the per layer objective function looks like? I don't get what he means by inserting the label into the input and what labels he is using per layer.
Obviously, this method is problematic if you have thousands of labels or if your network is not a classifier.
Deep dive tutorial for learning in a forward pass [1] [1] https://amassivek.github.io/sigprop
> There are many choices for a loss L (e.g. gradient, Hebbian) and optimizer (e.g. SGD, Momentum, ADAM). The output(), y, is detailed in step 4 below. I don't get it, don't all of those optimizers work via backprop?
Meta question on HN implementation: Why do sometimes submitting a previously submitted resource links automatically to the previous discussion, while other times is considered a new submission?
Earlier quoted context omitted.
Ad-hominems are not a particularly nice way to argue about correctness of a claim.
Maybe you know something but I don't, but I believe the comment you are replying to was a compliment and not a sarcastic dig.
Earlier quoted context omitted.
You have to start with toy models before scaling up.
Achieving 2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.
Earlier quoted context omitted.
You have to start with toy models before scaling up.
Achieving 2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.