Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who Invented Backpropagation?

people.idsia.ch

Post reply on HN
Print view View original

71–80 of 120 posts

Re: Who Invented Backpropagation?

#71

Post by dataflow »

throawayonthe wrote: post #20
dataflow wrote: post #12

Earlier quoted context omitted.

[flagged]

it's rude to show people your llm output

Wow, this is the first time I'm hearing such a thing. For clarity:

I pasted the output so a ton of people wouldn't repeat the same question to ChatGPT and burn a ton of CO2 to get the same answer.

I didn't paste the query since I didn't find it interesting.

And I didn't fact check because I didn't have the time. I was walking and had a few seconds to just do this on my phone.

Not sure how this was rude, I certainly didn't intend it to be...

Re: Who Invented Backpropagation?

#72

Post by mindcrime »

albertzeyer wrote: post #69
mindcrime wrote: post #67

Earlier quoted context omitted.

They are all in the article? Maybe they are. I'm not here to do a deep research project that involves reading every citation in that article. If it makes you feel better, pretend that what I said was instead: "I don't have all the relevant citations stored in my short-term memory right this second and I am not interested in writing a lengthy thesis to satisfy pedantic navel-gazers on HN." Or, if you really know some…

I thought that is what you mean when you said "a lot of interesting bits about the history of the development of neural networks (including backpropagation) can be found in the book Talking Nets", that there is some relevant reference to backprop which is missing here in the linked article. I don't really understand your negativity here, and what you are reading into my comment. I never asked you to do a research pro…

I thought that is what you mean when you said "a lot of interesting bits about the history of the development of neural networks (including backpropagation) can be found in the book Talking Nets", that there is some relevant reference to backprop which is missing here in the linked article.

Nah, I wasn't trying to imply that that book had anything more than the article, at least in regards to the back-prob question specifically. Just pointing it out as one more good resource for this kind of historical perspective.

I don't really understand your negativity here, and what you are reading into my comment. I never asked you to do a research project? I just thought you might know some other references which are not in the article. If you don't, fine.

No worries. I may be reacting more to a general HN meme than to you in particular. There's a certain brand of pedantry and obsessive nit-picking that is all too common here IMO. It grates on my nerves, so if I ever seem a little salty, it's probably because I thought somebody was doing that thing. It's all good. My apologies for the argumentative tone earlier.

Schmidhuber always try to be very careful to be very complete and exhaustive cite everything there is on some topic.

Agreed. That's one reason I don't get why people are always busting on Jurgen. For the most part, it seems that he can back up the claims he makes, and then some. I've heard plenty of people complain about him, but I'm not sure any of them have ever been able to state any particular sense in which he is actually wrong about anything. :-)

Re: Who Invented Backpropagation?

#73

Post by pretzellogician »

Kranar wrote: post #64
pretzellogician wrote: post #55

Earlier quoted context omitted.

That's a joke, right? Turning down community recognition and a million dollars to make an unclear statement about which category the prize was awarded in?

The argument had to do with honesty while your justification is that money and popularity are worth more than being honest. Now perhaps Hinton does deserve the award, but certainly it should not be because of the reasons you cite: money and popularity.

I said nothing of the sort. Being "honest" does not mean you have to give a middle finger to a panel that nominated you. The point of that Nobel was clearly recognition for their achievement; the category choice was mainly irrelevant.

Re: Who Invented Backpropagation?

#74

Post by Kranar »

pretzellogician wrote: post #73
Kranar wrote: post #64

Earlier quoted context omitted.

The argument had to do with honesty while your justification is that money and popularity are worth more than being honest. Now perhaps Hinton does deserve the award, but certainly it should not be because of the reasons you cite: money and popularity.

I said nothing of the sort. Being "honest" does not mean you have to give a middle finger to a panel that nominated you. The point of that Nobel was clearly recognition for their achievement; the category choice was mainly irrelevant.

No being honest does not mean that and had you said that I'd have no basis upon which to object to your comment.

You refuted an argument about being honest about accepting an award on the basis that the award pays a lot of money and grants one a great deal of popularity.

If your argument didn't involve money and popularity, then why did you choose those two specific criteria as the justification for accepting this award?

I want to be clear, I am not claiming that Dr. Hinton accepted the award in a dishonest manner or that he did it for money, I am simply refuting your position that money is a valid reason to disregard honesty for accepting a prestigious award.

Re: Who Invented Backpropagation?

#75

Post by 6gvONxR4sf7o »

My favorite take on this is that yes, in fact it is just the chain rule. The usual argument goes that automatic and symbolic differentiation are fundamentally different, so anything particularly old (pre-computers, for example) doesn't count as inventing back prop. But here's my favorite take on equivalences between AD and symbolic diff [0]. I wish there wasn't such importance placed on who invented it for stuff like this. Clearly, someone codifying backprop wasn't a bottleneck in making ML progress, so why's it get so much attention?

[0] https://emilien.ca/Notes/Notes/notes/1904.02990v4.pdf

Re: Who Invented Backpropagation?

#77

Post by brosco »

digikata wrote: post #25
mystraline wrote: post #3

> BP's modern version (also called the reverse mode of automatic differentiation) So... Automatic integration? Proportional, integrative, derivative. A PID loop sure sounds like what they're talking about.

There are large bodies of work for optimization of state space control theory that I strongly suspect as a lot of crossover for AI, and at least has very similar mathematical structure. e.g. optimization of state space control coefficients looks something like training a LLM matrix...

There is indeed a lot of crossover, and a lot of neural networks can be written in a state space form. The optimal control problem should be equivalent to training the weights, as you mention.

However, from what I have seen, this isn't really a useful way of reframing the problem. The optimal control problem is at least as hard, if not harder, than the original problem of training the neural network, and the latter has mature and performant software for doing it efficiently. That's not to say there isn't good software for optimal control, but it's a more general problem and therefore off-the-shelf solvers can't leverage the network structure very well.

Some researchers have made interesting theoretical connections like in neural ODEs, but even there the practicality is limited.

Re: Who Invented Backpropagation?

#78

Post by pretzellogician »

Kranar wrote: post #74
pretzellogician wrote: post #73

Earlier quoted context omitted.

I said nothing of the sort. Being "honest" does not mean you have to give a middle finger to a panel that nominated you. The point of that Nobel was clearly recognition for their achievement; the category choice was mainly irrelevant.

No being honest does not mean that and had you said that I'd have no basis upon which to object to your comment. You refuted an argument about being honest about accepting an award on the basis that the award pays a lot of money and grants one a great deal of popularity. If your argument didn't involve money and popularity, then why did you choose those two specific criteria as the justification for accepting this aw…

So we agree; you aren't claiming the award was accepted in a dishonest manner, and I never claimed anything about honesty being an issue. I simply found the idea of Hinton rejecting the award for the "honour of the Nobel [choice of category]" to be a silly idea.

Re: Who Invented Backpropagation?

#79

Post by epaulson »

dataflow wrote: post #71
throawayonthe wrote: post #20

Earlier quoted context omitted.

it's rude to show people your llm output

Wow, this is the first time I'm hearing such a thing. For clarity: I pasted the output so a ton of people wouldn't repeat the same question to ChatGPT and burn a ton of CO2 to get the same answer. I didn't paste the query since I didn't find it interesting. And I didn't fact check because I didn't have the time. I was walking and had a few seconds to just do this on my phone. Not sure how this was rude, I certainly d…

The 'it's rude to show your ai(llm) output' is a reference to this: https://distantprovince.by/posts/its-rude-to-show-ai-output-...
Post reply on HN