Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton publishes new deep learning algorithm

infoq.com

Post reply on HN
Print view View original

81–90 of 130 posts

Re: Geoffrey Hinton publishes new deep learning algorithm

#81

Post by jasonjmcghee »

electrograv wrote: post #73
whimsicalism wrote: post #72

Earlier quoted context omitted.

You have to start with toy models before scaling up.

Achieving 2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.

I haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range.

But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on.

For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.

Re: Geoffrey Hinton publishes new deep learning algorithm

#82

Post by whimsicalism »

electrograv wrote: post #73
whimsicalism wrote: post #72

Earlier quoted context omitted.

You have to start with toy models before scaling up.

Achieving 2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.

Any ML algorithm that already has tooling written, CUDA scripts, etc. to run it faster.

That said, I am also short-term bearish on backprop-free methods (although potentially long-term bullish).

Re: Geoffrey Hinton publishes new deep learning algorithm

#83

Post by whimsicalism »

jasonjmcghee wrote: post #81
electrograv wrote: post #73

Earlier quoted context omitted.

Achieving 2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.

I haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range. But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on. For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and…

> For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.

Making the assumption that weights/embeddings produced by a backprop-trained network are equally intelligible to a network also trained by backprop vs. one trained by this alternative method.

Re: Geoffrey Hinton publishes new deep learning algorithm

#84

Post by chamwislothe2nd »

nayroclade wrote: post #24
chamwislothe2nd wrote: post #22

Earlier quoted context omitted.

You're accusing one of the foundations of modern AI with either being a fraud, or incompetent. At best that seems short sighted, no?

You're obviously new to Hacker News :-D

Unfortunately, I've been here for a decade in one form or another. Every now and then someone writes something so pompous that I just can't help myself but post. Back to lurking now. Cheers!

Re: Geoffrey Hinton publishes new deep learning algorithm

#85

Post by gregschlom »

belter wrote: post #77

@dang Meta question on HN implementation: Why do sometimes submitting a previously submitted resource links automatically to the previous discussion, while other times is considered a new submission?

As far as I know it's a simple string match on the url. If the url is different (for example a new anchor tag is added) then it's considered a new submission.

Re: Geoffrey Hinton publishes new deep learning algorithm

#86

Post by habitue »

jasonjmcghee wrote: post #70

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure.

If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

Re: Geoffrey Hinton publishes new deep learning algorithm

#87

Post by LudwigNagasena »

habitue wrote: post #86
jasonjmcghee wrote: post #70

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

There is no shortage of paradigms that rip out backprop and deliver worse results.

Re: Geoffrey Hinton publishes new deep learning algorithm

#88

Post by tedsanders »

whimsicalism wrote: post #71
ipnon wrote: post #36

It’s incredible to think that dreams are just our brains generating training data, and lack of sleep causes us to overfit on our immediate surroundings.

We definitely do not know nearly enough to say anything like that with confidence. Most of the "training process" of our brain likely occurred prior to our birth in evolutionarily optimized structure of brain.

Unlikely. The human genome comprises only billions of bits, much of which is low-information repetition. The amount of information sensed over a lifetime is vastly greater. To sense less than a billion bits over a 30-year development period would imply less than one bit per second. We clearly perceive more than one bit per second. For this reason, it seems likely that more information comes from learning post-birth than is pre-conditioned by evolution pre-birth. (Though of course post-birth learning cannot take place without the fantastic foundation set by evolution.)

Re: Geoffrey Hinton publishes new deep learning algorithm

#89

Post by jxcole »

habitue wrote: post #86
jasonjmcghee wrote: post #70

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf , they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using a…

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy).

https://en.wikipedia.org/wiki/Capsule_neural_network

Re: Geoffrey Hinton publishes new deep learning algorithm

#90

Post by jasonjmcghee »

whimsicalism wrote: post #83
jasonjmcghee wrote: post #81

Earlier quoted context omitted.

I haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range. But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on. For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and…

> For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared. Making the assumption that weights/embeddings produced by a backprop-trained network are equally intelligible to a network also trained by backprop vs. one trained by this alternative method.

I have personally seen them used successfully with all kinds of classic ml algorithms (enets, tree-based, etc) that have nothing to do with back prop.
Post reply on HN