Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who Invented Backpropagation?

people.idsia.ch

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Who Invented Backpropagation?

#101

Post by jrowen »

pretzellogician wrote: post #78
Kranar wrote: post #74

Earlier quoted context omitted.

No being honest does not mean that and had you said that I'd have no basis upon which to object to your comment. You refuted an argument about being honest about accepting an award on the basis that the award pays a lot of money and grants one a great deal of popularity. If your argument didn't involve money and popularity, then why did you choose those two specific criteria as the justification for accepting this aw…

So we agree; you aren't claiming the award was accepted in a dishonest manner, and I never claimed anything about honesty being an issue. I simply found the idea of Hinton rejecting the award for the "honour of the Nobel [choice of category]" to be a silly idea.

Your indignation here seems a bit unwarranted. You definitely did bring the arguments about recognition and money into play and then basically gaslit someone for responding to them. You could have simply said that you misspoke.

Re: Who Invented Backpropagation?

#102

Post by drsopp »

Dylan16807 wrote: post #91
drsopp wrote: post #50

Earlier quoted context omitted.

> The point is that a reader may want to know that the text they're reading is something a human took the time to write themselves. The reader may also simply want information that helps them. > They just haven't commented. Yes, they did.

> The reader may also simply want information that helps them. The reader will generally want at least a cursory verification that it is information that helps, which dataflow didn't try to do. Especially when you're looking for specific documents and you don't check if the documents are real. (dataflow's third one doesn't appear to be.)

This I agree with completely.

Re: Who Invented Backpropagation?

#103

Post by vintermann »

dicroce wrote: post #33

Isn't it just kinda a natural thing once you have the chain rule?

Reverse mode differentiation? No, it can't be that natural since it took until 1970 to be proposed. But also in a sense basic (which you could also guess, since it was introduced in a MSc thesis).

Re: Who Invented Backpropagation?

#104

Post by seertaak »

pncnmnp wrote: post #8

I have a question that's bothered me for quite a while now. In 2018, Michael Jordan (UC Berkeley) wrote a rather interesting essay - https://medium.com/@mijordan3/artificial-intelligence-the-re... (Artificial Intelligence — The Revolution Hasn’t Happened Yet) In it, he stated the following: > Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now v…

Rumelhart et al wrote "Parallel Distributed Processing"; there's a chapter where he proves that the backprop algorithm maximizes "harmony", which is simply a different formulation of error minimization.

I remember reading this book enthusiastically back in the mid 90s. I don't recall struggling with the proof, it was fairly straightforward. (I was in senior high school year at the time.)

Re: Who Invented Backpropagation?

#105

Post by bgnn »

andyferris wrote: post #86
bgnn wrote: post #58

Earlier quoted context omitted.

So yeah, this was what I was thinking for a while. What about a more nonlinear estimator? Intuitively seems similar to me.

I believe the reason it works in nonlinear cases is that the derivative is “naturally linear” (to calculate the derivative, you are considering ever smaller regions where the cost function is approximately linear - exactly “how nonlinear” the cost function is elsewhere doesn’t play a role).

that makes a lot of sense actually. thank you.

Re: Who Invented Backpropagation?

#106

Post by cco »

drsopp wrote: post #16
pncnmnp wrote: post #8

I have a question that's bothered me for quite a while now. In 2018, Michael Jordan (UC Berkeley) wrote a rather interesting essay - https://medium.com/@mijordan3/artificial-intelligence-the-re... (Artificial Intelligence — The Revolution Hasn’t Happened Yet) In it, he stated the following: > Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now v…

Perhaps this: Henry J. Kelley (1960). Gradient Theory of Optimal Flight Paths. [1] https://claude.ai/public/artifacts/8e1dfe2b-69b0-4f2c-88f5-0...

Count another in the win column for the USA's heavy investment into basic sciences during the space race.

So sad to see the current state. Hopefully we can turn it around.

Re: Who Invented Backpropagation?

#108

Post by fooker »

eigenspace wrote: post #47
cs702 wrote: post #29

Whatever the facts, the OP comes across as sour grapes . The author, Jürgen Schmidhuber, believes Hopfield and Hinton did not deserve their Nobel Prize in Physics, and that Hinton, Bengio, and LeCun did not deserve their Turing Award. Evidently, many other scientists disagree, because both awards were granted in consultation with the scientific community. Schmidhuber's own work was, in fact, cited by the Nobel Prize…

For what it's worth, it's a very mainstream opinion in the physics community that Hinton did not at all deserve a nobel prize in physics for his work. But that's because his work, and wasnt impactful at all to the physics community

> wasnt impactful at all to the physics community

There are two reasons why Hinton got the prize.

A good majority of modern physics research depends on ML in some aspect. Look at the list of talks at any physics conference and count the number of talks that mention ML in the title.

And the 'physics community' has not produced any fundamental physics for a while. Look at the last several years of physics nobel prize. You can categorize the last ten years of prizes into two categories: engineering breakthroughs, and confirming important predictions. Both are impportant, but lacking fundamental physics breakthroughs, they are not clearly ahead in impact compared to ML.

Re: Who Invented Backpropagation?

#109

Post by cubefox »

dataflow wrote: post #81
epaulson wrote: post #79

Earlier quoted context omitted.

The 'it's rude to show your ai(llm) output' is a reference to this: https://distantprovince.by/posts/its-rude-to-show-ai-output-...

What is "this" exactly? Is it a well-known author or website? Or otherwise a reference that one should be familiar with? It looks like a random blog to me... with an opinion declared as fact that's quite easy to refute.

It got 321 points here: https://news.ycombinator.com/item?id=44617172

Re: Who Invented Backpropagation?

#110

Post by cubefox »

kypro wrote: post #60

I've always found it rather crazy that the power of backpropagation and artificial neural networks was doubted by AI researchers for so long. It's really only since the early 2010s that researchers started to take the field seriously. This is despite the core algorithm (backpropagation) being known for decades. I remember when I learnt about artificial neural networks at university in the late 00s my professors were…

Apart from backpropagation, the biggest improvements were probably changes in the network architecture. Standard feed-forward MPCs are fairly inefficient. Then there were architectures like CNNs, LSTMs, Transformers. There were also improvements in the activation function and in the gradient descent method (AdamW), but I'm not sure whether these had a substantial impact like CNNs or Transformers. Another factor was training on GPUs.
Post reply on HN