Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who Invented Backpropagation?

people.idsia.ch

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Who Invented Backpropagation?

#111

Post by cubefox »

cs702 wrote: post #29

Whatever the facts, the OP comes across as sour grapes . The author, Jürgen Schmidhuber, believes Hopfield and Hinton did not deserve their Nobel Prize in Physics, and that Hinton, Bengio, and LeCun did not deserve their Turing Award. Evidently, many other scientists disagree, because both awards were granted in consultation with the scientific community. Schmidhuber's own work was, in fact, cited by the Nobel Prize…

> Evidently, many other scientists disagree, because both awards were granted in consultation with the scientific community.

That's not a good argument. They do in fact sometimes give awards with which the "scientific community" disagrees. Schmidhuber actually gave object level arguments on why the official justification for the Turing award contained substantial errors.

Re: Who Invented Backpropagation?

#112

Post by cutlilacs »

fooker wrote: post #108
eigenspace wrote: post #47

Earlier quoted context omitted.

For what it's worth, it's a very mainstream opinion in the physics community that Hinton did not at all deserve a nobel prize in physics for his work. But that's because his work, and wasnt impactful at all to the physics community

> wasnt impactful at all to the physics community There are two reasons why Hinton got the prize. A good majority of modern physics research depends on ML in some aspect. Look at the list of talks at any physics conference and count the number of talks that mention ML in the title. And the 'physics community' has not produced any fundamental physics for a while. Look at the last several years of physics nobel prize.…

I’m confused why confirming important predictions is considered less impactful than ML in physics. Isn’t experimental confirmation exactly what’s required for a Nobel Prize?

Re: Who Invented Backpropagation?

#113

Post by cutlilacs »

_fizz_buzz_ wrote: post #59
Jensson wrote: post #51

Earlier quoted context omitted.

You don't think the people who invented the chain rule understood what it means?

Obviously, Newton and Leibniz and many other Mathematicians (and other people) understood the chain rule before back propagation. But unfortunately I am very far from a Newton or Leibniz, so it took me a lot longer to grasp why the chain rule is the way it is. And back propagation just made it click for me. I was really just talking about me personally.

What insight did you gain from back propagation that you didn't have from just the formula of the chain rule?

Re: Who Invented Backpropagation?

#114

Post by fooker »

cutlilacs wrote: post #112
fooker wrote: post #108

Earlier quoted context omitted.

> wasnt impactful at all to the physics community There are two reasons why Hinton got the prize. A good majority of modern physics research depends on ML in some aspect. Look at the list of talks at any physics conference and count the number of talks that mention ML in the title. And the 'physics community' has not produced any fundamental physics for a while. Look at the last several years of physics nobel prize.…

I’m confused why confirming important predictions is considered less impactful than ML in physics. Isn’t experimental confirmation exactly what’s required for a Nobel Prize?

Experimental confirmation of X makes X great physics and X worthy of a nobel prize, not the engineering setup needed for the experimental confirmation.

The setup by itself can also general technique that is useful beyond confirming one thing (example LIGO). But then, ML is itself is a more general technique that has enabled a lot more new physics than one new experiment.

Re: Who Invented Backpropagation?

#115

Post by cutlilacs »

fooker wrote: post #114
cutlilacs wrote: post #112

Earlier quoted context omitted.

I’m confused why confirming important predictions is considered less impactful than ML in physics. Isn’t experimental confirmation exactly what’s required for a Nobel Prize?

Experimental confirmation of X makes X great physics and X worthy of a nobel prize, not the engineering setup needed for the experimental confirmation. The setup by itself can also general technique that is useful beyond confirming one thing (example LIGO). But then, ML is itself is a more general technique that has enabled a lot more new physics than one new experiment.

I would couple the Experiment and the theory together, and treat them both deserving of the prize, but not sure how it works in practice. As for the general technique of ML, sure, it's important but it seems to me that it's a tool that can be used in Physics, and the specific implementation/use-case is the actual thing that's noteworthy, not the general tool. I wouldn't consider a new mathematical theorem by itself to be physics and deserving of a physics prize, I view general ML the same way.

Re: Who Invented Backpropagation?

#116

Post by blt »

brosco wrote: post #77
digikata wrote: post #25

Earlier quoted context omitted.

There are large bodies of work for optimization of state space control theory that I strongly suspect as a lot of crossover for AI, and at least has very similar mathematical structure. e.g. optimization of state space control coefficients looks something like training a LLM matrix...

There is indeed a lot of crossover, and a lot of neural networks can be written in a state space form. The optimal control problem should be equivalent to training the weights, as you mention. However, from what I have seen, this isn't really a useful way of reframing the problem. The optimal control problem is at least as hard, if not harder, than the original problem of training the neural network, and the latter h…

Yes, in most cases the reduction of supervised learning to optimal control is not interesting.

We can also reduce supervised learning to reinforcement learning, but that doesn't mean we should use RL algorithms to do supervised learning.

We can also reduce sorting a list of integers to SAT, but that doesn't mean we should use a SAT solver to sort lists of integers.

Re: Who Invented Backpropagation?

#117

Post by fooker »

cutlilacs wrote: post #115
fooker wrote: post #114

Earlier quoted context omitted.

Experimental confirmation of X makes X great physics and X worthy of a nobel prize, not the engineering setup needed for the experimental confirmation. The setup by itself can also general technique that is useful beyond confirming one thing (example LIGO). But then, ML is itself is a more general technique that has enabled a lot more new physics than one new experiment.

I would couple the Experiment and the theory together, and treat them both deserving of the prize, but not sure how it works in practice. As for the general technique of ML, sure, it's important but it seems to me that it's a tool that can be used in Physics, and the specific implementation/use-case is the actual thing that's noteworthy, not the general tool. I wouldn't consider a new mathematical theorem by itself t…

Ideally this would be coupled like you say, but often in physics these are increasingly further apart, often by several decades.

And a large number of predictions being made now are unlikely to be ever confirmed.

Re: Who Invented Backpropagation?

#118

Post by _fizz_buzz_ »

cutlilacs wrote: post #113
_fizz_buzz_ wrote: post #59

Earlier quoted context omitted.

Obviously, Newton and Leibniz and many other Mathematicians (and other people) understood the chain rule before back propagation. But unfortunately I am very far from a Newton or Leibniz, so it took me a lot longer to grasp why the chain rule is the way it is. And back propagation just made it click for me. I was really just talking about me personally.

What insight did you gain from back propagation that you didn't have from just the formula of the chain rule?

What clicked for me was drawing the chain rule as a graph. When I was in school I just applied the chain rule without thinking about it. I really didn't mean this to be some deep insight or anything. Just an anecdotal comment.

Re: Who Invented Backpropagation?

#119

Post by qarl »

uoaei wrote: post #34
qarl wrote: post #14

Earlier quoted context omitted.

From the article: Some ask: "Isn't backpropagation just the chain rule of Leibniz (1676) [LEI07-10] & L'Hopital (1696)?" No, it is the efficient way of applying the chain rule to big networks with differentiable nodes—see Sec. XII of [T22][DLH]). (There are also many inefficient ways of doing this.) It was not published until 1970 [BP1].

The article says that but it's overcomplicating to the point of being actually wrong. You could, I suppose, argue that the big innovation is the application of vectorization to the chain rule (by virtue of the matmul-based architecture of your usual feedforward network) which is a true combination of two mathematical technologies. But it feels like this and indeed most "innovations" in ML is only considered as such d…

The real metric is whether anyone remembers it in 100 years. Any other discussion just comes off as petty.

Re: Who Invented Backpropagation?

#120

Post by cutlilacs »

_fizz_buzz_ wrote: post #118
cutlilacs wrote: post #113

Earlier quoted context omitted.

What insight did you gain from back propagation that you didn't have from just the formula of the chain rule?

What clicked for me was drawing the chain rule as a graph. When I was in school I just applied the chain rule without thinking about it. I really didn't mean this to be some deep insight or anything. Just an anecdotal comment.

Ah makes sense, I was thinking there was some deeper insight I was missing. Thanks!
Post reply on HN