Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Who Invented Backpropagation?

people.idsia.ch

Post reply on HN
Print view View original

31–40 of 120 posts

Re: Who Invented Backpropagation?

#31

Post by pjbk »

duped wrote: post #18
pncnmnp wrote: post #8

I have a question that's bothered me for quite a while now. In 2018, Michael Jordan (UC Berkeley) wrote a rather interesting essay - https://medium.com/@mijordan3/artificial-intelligence-the-re... (Artificial Intelligence — The Revolution Hasn’t Happened Yet) In it, he stated the following: > Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now v…

They're probably talking about Kalman Filters (1961) and LMS filters (1960).

To be fair, any multivariable regulator or filter (estimator) that has a quadratic component (LQR/LQE) will naturally yield a solution similar to backpropagation when an iterative algorithm is used to optimize its cost or error function through a differentiable tangent space.

Re: Who Invented Backpropagation?

#34

Post by uoaei »

qarl wrote: post #14
dudu24 wrote: post #11

It's just an application of the chain rule. It's not interesting to ask who invented it.

From the article: Some ask: "Isn't backpropagation just the chain rule of Leibniz (1676) [LEI07-10] & L'Hopital (1696)?" No, it is the efficient way of applying the chain rule to big networks with differentiable nodes—see Sec. XII of [T22][DLH]). (There are also many inefficient ways of doing this.) It was not published until 1970 [BP1].

The article says that but it's overcomplicating to the point of being actually wrong. You could, I suppose, argue that the big innovation is the application of vectorization to the chain rule (by virtue of the matmul-based architecture of your usual feedforward network) which is a true combination of two mathematical technologies. But it feels like this and indeed most "innovations" in ML is only considered as such due to brainrot derived from trying to take maximal credit for minimal work (i.e., IP).

Re: Who Invented Backpropagation?

#35

Post by drsopp »

microtonal wrote: post #30
drsopp wrote: post #21

Earlier quoted context omitted.

Why?

Because it is terribly low-effort. People are here for interesting and insightful discussions with other humans. If they were interested in unverified LLM output… they would ask an LLM?

Who cares if it is low effort? I got lots of upvotes for my link to Claude about this, and pncnmnp seems happy. The downvoted comment from ChatGPT was maybe a bit spammy?

Re: Who Invented Backpropagation?

#38

Post by _fizz_buzz_ »

bjornsing wrote: post #37

The chain rule was explored by Gottfried Wilhelm Leibniz and Isaac Newton in the 17th century. Either of them would have ”invented” backpropagation in an instant. It’s obvious.

Funny enough. For me it was the other way around. I always knew how to compute the chain rule. But really only understood what the chain rule means when I read up on what back propagation was.

Re: Who Invented Backpropagation?

#39

Post by aeonik »

throawayonthe wrote: post #20
dataflow wrote: post #12

Earlier quoted context omitted.

[flagged]

it's rude to show people your llm output

I don't think it's rude, it saves me from having to come up with my own prompt and wade through the back and forth to get useful insight from the LLMs, also saves me from spending my tokens.

Also, I quite love it when people clearly demarcate which part of their content came from an LLM, and specifies which model.

The little citation carries a huge amount of useful information.

The folks who don't like AI should like it too, as they can easily filter the content.

Re: Who Invented Backpropagation?

#40

Post by icelancer »

cs702 wrote: post #29

Whatever the facts, the OP comes across as sour grapes . The author, Jürgen Schmidhuber, believes Hopfield and Hinton did not deserve their Nobel Prize in Physics, and that Hinton, Bengio, and LeCun did not deserve their Turing Award. Evidently, many other scientists disagree, because both awards were granted in consultation with the scientific community. Schmidhuber's own work was, in fact, cited by the Nobel Prize…

Didn't click the article, came straight to the comments thinking "I bet it's Schmidhuber being salty."

Some things never change.

Post reply on HN