Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton publishes new deep learning algorithm

infoq.com

Post reply on HN
Print view View original

111–120 of 130 posts

Re: Geoffrey Hinton publishes new deep learning algorithm

#111

Post by davidgay »

snowpid wrote: post #50
mlajtos wrote: post #40

Earlier quoted context omitted.

We tend to start hallucinating when we don't have enough sleep. So generating training data is necessary, but way safer when our muscles are turned off.

in addition: During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep. My guess, this sleep pattern is better for learning.

> During pre electricity time humen woke up after 4 hours sleep, got awake for some time and then continue to sleep.

The confusing thing with this claim is what did people actually do during this time, given bad (and expensive!) lighting only?

Re: Geoffrey Hinton publishes new deep learning algorithm

#112

Post by tehsauce »

LudwigNagasena wrote: post #87
habitue wrote: post #86

Earlier quoted context omitted.

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

There is no shortage of paradigms that rip out backprop and deliver worse results.

This is so true! But we should keep trying :)

Re: Geoffrey Hinton publishes new deep learning algorithm

#113

Post by whimsicalism »

levesque wrote: post #106
LudwigNagasena wrote: post #87

Earlier quoted context omitted.

There is no shortage of paradigms that rip out backprop and deliver worse results.

Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.

I think it can be simultaneously true that things like this should be tested with toy models we wouldn't expect to do great on CIFAR and also that we shouldn't expect exceptional results just because this person is already famous.

Re: Geoffrey Hinton publishes new deep learning algorithm

#114

Post by Animats »

sva_ wrote: post #92

I found this paragraph from the paper very interesting: > 7 The relevance of FF to analog hardware > An energy efficient way to multiply an activity vector by a weight matrix is to implement activities as voltages and weights as conductances. Their products, per unit time, are charges which add themselves. This seems a lot more sensible than driving transistors at high power to model the individual bits in the digita…

There's been unhappiness in some quarters that back propagation doesn't seem to be something that biology does. That may be part of the motivation here.

Re: Geoffrey Hinton publishes new deep learning algorithm

#115

Post by habitue »

jxcole wrote: post #89
habitue wrote: post #86

Earlier quoted context omitted.

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy). https://en.wikipedia.org/wiki/Capsule_neural_network

I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks

Re: Geoffrey Hinton publishes new deep learning algorithm

#117

Post by oneoff786 »

tedsanders wrote: post #88
whimsicalism wrote: post #71

Earlier quoted context omitted.

We definitely do not know nearly enough to say anything like that with confidence. Most of the "training process" of our brain likely occurred prior to our birth in evolutionarily optimized structure of brain.

Unlikely. The human genome comprises only billions of bits, much of which is low-information repetition. The amount of information sensed over a lifetime is vastly greater. To sense less than a billion bits over a 30-year development period would imply less than one bit per second. We clearly perceive more than one bit per second. For this reason, it seems likely that more information comes from learning post-birth t…

> The human genome comprises only billions of bits, much of which is low-information repetition.

We constantly find out that certain things are actually really important even though we thought it was junk. Recall that our best ability to test Genes is by knocking them out one by one and trying to observe the effect

The brain is comprised of many extremely specialized sub systems and formulas for generating knowledge. We don’t know English at birth, sure, but we do have a language processing capability. The training baked into the brain is a level of abstraction higher, establishing frameworks to learn other things. It may not be as storage data heavy, but it’s much harder to arrive at and is the bulk of the learning process (learning to learn)

Re: Geoffrey Hinton publishes new deep learning algorithm

#118

Post by oneoff786 »

UncleEntity wrote: post #57
LudwigNagasena wrote: post #32

Earlier quoted context omitted.

It performs worse than baсkprop.

So the initial “hey, this might be a good idea” implementation performs slightly worse than something that has had literally billions of dollars thrown at it?

Very significantly worse.

Re: Geoffrey Hinton publishes new deep learning algorithm

#119

Post by mdp2021 »

danuker wrote: post #97
gregschlom wrote: post #85

Earlier quoted context omitted.

As far as I know it's a simple string match on the url. If the url is different (for example a new anchor tag is added) then it's considered a new submission.

If you click on "past" under this submission, you see two identical URLs: https://hn.algolia.com/?query=Geoffrey%20Hinton%20publishes%...

Which is odd, because I checked in the minutes following the submission and I remember "[past]" did not return anything.

Re: Geoffrey Hinton publishes new deep learning algorithm

#120

Post by NHQ »

parpfish wrote: post #61
bayesian_horse wrote: post #21

Earlier quoted context omitted.

Fact: Geoffrey Hinton has discovered how the brain works. Every few years actually.

yeah, whatever happened to capsule networks?

Capsule networks were conceptually an early attempt at transformers.
Post reply on HN