Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton publishes new deep learning algorithm

infoq.com

Post reply on HN
Print view View original

121–130 of 130 posts

Re: Geoffrey Hinton publishes new deep learning algorithm

#121

Post by Nevermark »

jasonjmcghee wrote: post #70

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…

The best context to view the paper, is as part of an algorithm search.

Until the brain's algorithm is "solved", half steps are important. We need as many alternate half steps as we can find until one or more lead to a better understanding of the brain. (And potentially, better than backdrop efficiency or results.)

Re: Geoffrey Hinton publishes new deep learning algorithm

#122

Post by NHQ »

Hinton's networks become the neuron of novel networks. It is important to know that these types of weights don't learn features, they map a compressed representation of the learned info, which is the input. Classification through error correction. That is actually what labels do for supervised learning (IOW they learn many ways to represent the label, and that is what the weights are). Modern AI do that plus learn features, but the weights are nevertheless a representation of what was learned, plus a fancy way to encode and decode into that domain.

What Hinton and Deepmind will do is use neural-network learned-data, or perhaps the weights, as input to this kind of network. In other words, the output of another NN is labeled a priori, ergo you can use it "unsupervised" networks, which this research expounds. This will allow them to cook the input network into a specific dish, by labels even. Now give me my phd.

edit: edit

Re: Geoffrey Hinton publishes new deep learning algorithm

#123

Post by hgsgm »

habitue wrote: post #115
jxcole wrote: post #89

Earlier quoted context omitted.

This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy). https://en.wikipedia.org/wiki/Capsule_neural_network

I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks

But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.

Re: Geoffrey Hinton publishes new deep learning algorithm

#124

Post by deanCommie »

hgsgm wrote: post #123
habitue wrote: post #115

Earlier quoted context omitted.

I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks

But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.

The backpropagation paper was published in 1986.

It took >20 years for it to be right.

Maybe we ought to give this one some time?

Re: Geoffrey Hinton publishes new deep learning algorithm

#125

Post by habitue »

hgsgm wrote: post #123
habitue wrote: post #115

Earlier quoted context omitted.

I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks

But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.

Specifically, tenure is to remove the pressure that "you'd better be right" so professors are free to take meandering tangents through the solution space that don't seem like they'll pay off immediately.

The failure mode of tenure is that the professor just rests on their past accomplishments and doesn't do anything. That's a risk the system takes. In this case though, Geoff Hinton is doing everything right: he's not only not sitting around doing nothing, he's actively trying to obsolete the paradigm he helped usher in, just in case there is a better option out there. I think that's admirable

Re: Geoffrey Hinton publishes new deep learning algorithm

#126

Post by kwooding »

deanCommie wrote: post #124
hgsgm wrote: post #123

Earlier quoted context omitted.

But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.

The backpropagation paper was published in 1986. It took >20 years for it to be right. Maybe we ought to give this one some time?

Not sure what you mean by >20 years to be right. I built and trained a 3-layer back-propagating neural net to do OCR on an Apple 2 in 1989 based on that paper. Admittedly, just the 26 upper case characters. But it clearly worked better than the alternatives.

Re: Geoffrey Hinton publishes new deep learning algorithm

#127

Post by snowpid »

davidgay wrote: post #111
snowpid wrote: post #50

Earlier quoted context omitted.

in addition: During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep. My guess, this sleep pattern is better for learning.

> During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep. The confusing thing with this claim is what did people actually do during this time, given bad (and expensive!) lighting only?

Thinking, talking to peers, Smartphones are a new invention and night was a dangerous time for a long time.

Re: Geoffrey Hinton publishes new deep learning algorithm

#128

Post by mlajtos »

snowpid wrote: post #50
mlajtos wrote: post #40

Earlier quoted context omitted.

We tend to start hallucinating when we don't have enough sleep. So generating training data is necessary, but way safer when our muscles are turned off.

in addition: During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep. My guess, this sleep pattern is better for learning.

It's called biphasic sleep for people that want to read up on it.

> My guess, this sleep pattern is better for learning.

That might be true. One of the techniques to induce lucid dreaming works similarly – sleep for 4-5 hours, wake up, stay awake for 15-60mins then go back to sleep. It's called "wake back to bed" technique. Many lucid dreamers report increased capacity for learning in dreams.

Re: Geoffrey Hinton publishes new deep learning algorithm

#129

Post by yobbo »

jasonjmcghee wrote: post #70

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…

This is not a benchmark of some model on cifar-10, it's a benchmark of the training algorithm.

But, model size and complexity also matters. MLP with backprop gets about 63% on cifar-10, for various reasons. So achieving 59% accuracy means this algorithm is about 93% as good as backprop in this case.

However, 63% accuracy on cifar-10 can be achieved with two (maybe three) layers IIRC. The output is a 10-way classifier, which is handled in one layer. If the output requires multi-layer transformations, then gradients need to be back-propagated.

As long as the batch activation vectors are trained to max separation (or orthogonality or whatever) at each layer, one output layer can match them to labels. But this is unlikely in problems where the output is more "transformed or complicated".

Re: Geoffrey Hinton publishes new deep learning algorithm

#130

Post by emilecourth »

There's an open source implementation of the paper in pytorch https://github.com/nebuly-ai/nebullvm/tree/main/apps/acceler... by @diegofiori_

He also wrote an interesting thread on the memory usage of this algo versus backprop https://twitter.com/diegofiori_/status/1605242573311709184?s...

Post reply on HN