Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton publishes new deep learning algorithm

infoq.com

Post reply on HN
Print view View original

91–100 of 130 posts

Re: Geoffrey Hinton publishes new deep learning algorithm

#91

Post by PartiallyTyped »

chamwislothe2nd wrote: post #84
nayroclade wrote: post #24

Earlier quoted context omitted.

You're obviously new to Hacker News :-D

Unfortunately, I've been here for a decade in one form or another. Every now and then someone writes something so pompous that I just can't help myself but post. Back to lurking now. Cheers!

If we can't publicly scrutinize people who have great sway in the industry, what does that say about us as a research community?

The fact that I argued why I found it bogus based on well established principles, and I get shitted on by people who by all means have provided nothing to this conversation and except suppressing criticism or throwing ad-hominems should tell all about the quality of discourse.

Dismissing criticism, not by arguments, but by the mere name of the person does a disservice to everyone.

If the research can't stand on its own, independent of the author, then it is not good research.

Re: Geoffrey Hinton publishes new deep learning algorithm

#92

Post by sva_ »

I found this paragraph from the paper very interesting:

> 7 The relevance of FF to analog hardware

> An energy efficient way to multiply an activity vector by a weight matrix is to implement activities as voltages and weights as conductances. Their products, per unit time, are charges which add themselves. This seems a lot more sensible than driving transistors at high power to model the individual bits in the digital representation of a number and then performing O(n^2) single bit operations to multiply two n-bit numbers together. Unfortunately, it is difficult to implement the backpropagation procedure in an equally efficient way, so people have resorted to using A-to-D converters and digital computations for computing gradients (Kendall et al., 2020). The use of two forward passes instead of a forward and a backward pass should make these A-to-D converters unnecessary.

It was my impression that it is difficult to properly isolate an electronic system to use voltages in this way (hence computers sort of "cut" voltages into bits 0/1 using a step function).

Have these limitations been overcome or do they not matter as much, as neural networks can work with more fuzzy data?

Interesting to imagine such a processor though.

Re: Geoffrey Hinton publishes new deep learning algorithm

#93

Post by drdeca »

belter wrote: post #77

@dang Meta question on HN implementation: Why do sometimes submitting a previously submitted resource links automatically to the previous discussion, while other times is considered a new submission?

I believe one factor is the amount of time between the submissions.

Re: Geoffrey Hinton publishes new deep learning algorithm

#94

Post by dimatura »

ekleraki wrote: post #52
dimatura wrote: post #34

Earlier quoted context omitted.

Yeah, no. Reading the paper I don't really see anything but a superficial resemblance to stacking. Hinton was active back when Wolpert introduced stacking and I'm fairly sure he is aware of it. If anything it much more closely resembles his own prior work in Boltzmann machines, unsurprisingly (and which he cites), or even his prior work on capsules. I don't know if this will really pan out into anything that differen…

A single linear layer is for all intents and purposes equivalent to running an ensemble of linear estimators. By disallowing gradients to flow between two layers A, B, computing (B . f . A)(x) with f being a non linearity, the second layer is an ensemble of linear estimators of the outputs of the first, and for all intents and purposes, the output of (f.A)(x) is just preprocessing for B. Since gradients don't flow fr…

I'm not sure what the main point is here. The paper is definitely sketchy on details, and the main idea is definitely simple enough to resemble a lot of other work. I wouldn't be surprised if someone (maybe a certain Swiss researcher) comes out and says, actually, this is the same as this other paper from the early 90s. If you squint hard enough a lot of ideas (especially simple ones) can be seen as being the same as other, older ideas. I'm not too interested in splitting those hairs, really. I'm more curious on whether this eventually leads to something that sets it apart from the SOTA in some interesting way.

Re: Geoffrey Hinton publishes new deep learning algorithm

#95

Post by chamwislothe2nd »

PartiallyTyped wrote: post #91
chamwislothe2nd wrote: post #84

Earlier quoted context omitted.

Unfortunately, I've been here for a decade in one form or another. Every now and then someone writes something so pompous that I just can't help myself but post. Back to lurking now. Cheers!

If we can't publicly scrutinize people who have great sway in the industry, what does that say about us as a research community? The fact that I argued why I found it bogus based on well established principles, and I get shitted on by people who by all means have provided nothing to this conversation and except suppressing criticism or throwing ad-hominems should tell all about the quality of discourse. Dismissing cr…

We are all looking forward to your research paper that disproves his claims. Or you know, any proof.

Re: Geoffrey Hinton publishes new deep learning algorithm

#96

Post by hiddencost »

electrograv wrote: post #73
whimsicalism wrote: post #72

Earlier quoted context omitted.

You have to start with toy models before scaling up.

Achieving 2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.

Hinton is doing basic science, not ML, here. Given who he is, trying to move the needle on traditional benchmarks would be a waste of his time and skills.

If he invents the new back propagation, an army of grad students can turn his ideas into the future. Like they've done for the last 15 years.

He's posting incremental work towards rethinking the field. It's pretty interesting stuff.

Edit: grammar

Re: Geoffrey Hinton publishes new deep learning algorithm

#97

Post by danuker »

gregschlom wrote: post #85
belter wrote: post #77

@dang Meta question on HN implementation: Why do sometimes submitting a previously submitted resource links automatically to the previous discussion, while other times is considered a new submission?

As far as I know it's a simple string match on the url. If the url is different (for example a new anchor tag is added) then it's considered a new submission.

If you click on "past" under this submission, you see two identical URLs:

https://hn.algolia.com/?query=Geoffrey%20Hinton%20publishes%...

Re: Geoffrey Hinton publishes new deep learning algorithm

#98

Post by PartiallyTyped »

dimatura wrote: post #94
ekleraki wrote: post #52

Earlier quoted context omitted.

A single linear layer is for all intents and purposes equivalent to running an ensemble of linear estimators. By disallowing gradients to flow between two layers A, B, computing (B . f . A)(x) with f being a non linearity, the second layer is an ensemble of linear estimators of the outputs of the first, and for all intents and purposes, the output of (f.A)(x) is just preprocessing for B. Since gradients don't flow fr…

I'm not sure what the main point is here. The paper is definitely sketchy on details, and the main idea is definitely simple enough to resemble a lot of other work. I wouldn't be surprised if someone (maybe a certain Swiss researcher) comes out and says, actually, this is the same as this other paper from the early 90s. If you squint hard enough a lot of ideas (especially simple ones) can be seen as being the same as…

My claim is that this work is simply worse ensembles wrapped in a biologically inspired claims, and that arguments made in support of it by the author compared to other approaches are simply not sound.

By looking at it through that perspective, the issues with the approach become evident, and are fundamental in my opinion.

Re: Geoffrey Hinton publishes new deep learning algorithm

#99

Post by PartiallyTyped »

chamwislothe2nd wrote: post #95
PartiallyTyped wrote: post #91

Earlier quoted context omitted.

If we can't publicly scrutinize people who have great sway in the industry, what does that say about us as a research community? The fact that I argued why I found it bogus based on well established principles, and I get shitted on by people who by all means have provided nothing to this conversation and except suppressing criticism or throwing ad-hominems should tell all about the quality of discourse. Dismissing cr…

We are all looking forward to your research paper that disproves his claims. Or you know, any proof.

I argued for it and all I got was downvoted without criticism of the substance of my arguments, only ad-hominems and fallacies.

If you can point to _fundamental_ criticism of my arguments, and not fallacies or attacks, I'd be more than happy to discuss them.

Re: Geoffrey Hinton publishes new deep learning algorithm

#100

Post by belter »

ekleraki wrote: post #67
randomdata wrote: post #59

Earlier quoted context omitted.

A lot can change in 60 days.

They do have a master's degree according to the post. The claims made are not that deep for researchers.

The use of the word masters is now considered not cool according to Stanford... :-)
Post reply on HN