Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Geoffrey Hinton publishes new deep learning algorithm

infoq.com

Post reply on HN
Print view View original

101–110 of 130 posts

Re: Geoffrey Hinton publishes new deep learning algorithm

#101

Post by btown »

sva_ wrote: post #92

I found this paragraph from the paper very interesting: > 7 The relevance of FF to analog hardware > An energy efficient way to multiply an activity vector by a weight matrix is to implement activities as voltages and weights as conductances. Their products, per unit time, are charges which add themselves. This seems a lot more sensible than driving transistors at high power to model the individual bits in the digita…

Photonic/optical neural networks are an interesting related area of research, using light interference to implement convolution and other operations without (I believe?) needing a bitwise representation of intensity.

https://www.nature.com/articles/s41467-020-20719-7

https://opg.optica.org/optica/fulltext.cfm?uri=optica-5-7-86...

Re: Geoffrey Hinton publishes new deep learning algorithm

#102

Post by PartiallyTyped »

belter wrote: post #100
ekleraki wrote: post #67

Earlier quoted context omitted.

They do have a master's degree according to the post. The claims made are not that deep for researchers.

The use of the word masters is now considered not cool according to Stanford... :-)

It's not like I have been doing many things outside reading papers and books over the past few years... The post that person used as an ad-hominem even says so.

Re: Geoffrey Hinton publishes new deep learning algorithm

#103

Post by godelski »

bitL wrote: post #42
harveywi wrote: post #33

Earlier quoted context omitted.

Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.

Jürgen did it all before in the 80s, however it was never translated to English so Geoffrey could happily reinvent it.

Jürgen invented AGI in the early 90s but someone pressed the red button on his website and it committed suicide.

Re: Geoffrey Hinton publishes new deep learning algorithm

#104

Post by CuriouslyC »

sva_ wrote: post #92

I found this paragraph from the paper very interesting: > 7 The relevance of FF to analog hardware > An energy efficient way to multiply an activity vector by a weight matrix is to implement activities as voltages and weights as conductances. Their products, per unit time, are charges which add themselves. This seems a lot more sensible than driving transistors at high power to model the individual bits in the digita…

The small deltas resulting from electrical noise generally aren't an issue for probabilistic computations. Interestingly, people have quantized many large DL models down to 8/16 bits, and accuracy reduction is often on the order of 2-5%. Additionally, adding random noise to weights during training tends to act as a form of regularization.

Re: Geoffrey Hinton publishes new deep learning algorithm

#105

Post by duvenaud »

goethes_kind wrote: post #26

I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…

Perhaps a better place to find algorithmic details is this related paper, also with Hinton as a co-author, which implements similar ideas in more standard networks:

Scaling Forward Gradient With Local Losses Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton https://arxiv.org/abs/2210.03310

and has code: https://github.com/google-research/google-research/tree/mast...

Re: Geoffrey Hinton publishes new deep learning algorithm

#106

Post by levesque »

LudwigNagasena wrote: post #87
habitue wrote: post #86

Earlier quoted context omitted.

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

There is no shortage of paradigms that rip out backprop and deliver worse results.

Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.

Re: Geoffrey Hinton publishes new deep learning algorithm

#107

Post by godelski »

bjourne wrote: post #44
goethes_kind wrote: post #26

I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same lab…

Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of…

> because the idea is trivial in hindsight (always is) so publishing fast is important.

Unfortunately I've also seen papers get rejected because their idea was "trivial", yet no one had thought of it before. Hinton has an edge here though.

Re: Geoffrey Hinton publishes new deep learning algorithm

#109

Post by chamwislothe2nd »

PartiallyTyped wrote: post #99
chamwislothe2nd wrote: post #95

Earlier quoted context omitted.

We are all looking forward to your research paper that disproves his claims. Or you know, any proof.

I argued for it and all I got was downvoted without criticism of the substance of my arguments, only ad-hominems and fallacies. If you can point to _fundamental_ criticism of my arguments, and not fallacies or attacks, I'd be more than happy to discuss them.

The hard dismissal with 'No' is likely why you got down voted. I am not able to do that.

With that kind of tonal promise, especially considering the source you are dismissing outright is important in their field, you have to show, not just tell.

If you just left that No out, and gave room for the chance that you are wrong, people wouldn't downvote, they'd upvote. People like to hear smart arguments. No one wants to hear outward dismissal. Especially of known experts.

Post reply on HN