Earlier quoted context omitted.
[flagged]
it's rude to show people your llm output
Why?
Who Invented Backpropagation?
21–30 of 120 posts
Re: Who Invented Backpropagation?
#22this fight has become legendary and infamous
Re: Who Invented Backpropagation?
#23I have a question that's bothered me for quite a while now. In 2018, Michael Jordan (UC Berkeley) wrote a rather interesting essay - https://medium.com/@mijordan3/artificial-intelligence-the-re... (Artificial Intelligence — The Revolution Hasn’t Happened Yet) In it, he stated the following: > Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now v…
Perhaps this: Henry J. Kelley (1960). Gradient Theory of Optimal Flight Paths. [1] https://claude.ai/public/artifacts/8e1dfe2b-69b0-4f2c-88f5-0...
Thanks! This might be it. I looked up Henry J. Kelley on Wikipedia, and in the notes I found a citation to this paper from Stuart Dreyfus (Berkeley): "Artificial Neural Networks, Back Propagation and the Kelley-Bryson Gradient Procedure" (https://gwern.net/doc/ai/nn/1990-dreyfus.pdf).
I am still going through it, but the latter is quite interesting!
Re: Who Invented Backpropagation?
#24this fight has become legendary and infamous, and also pops up on HN every 2-3 years
Re: Who Invented Backpropagation?
#25> BP's modern version (also called the reverse mode of automatic differentiation) So... Automatic integration? Proportional, integrative, derivative. A PID loop sure sounds like what they're talking about.
There are large bodies of work for optimization of state space control theory that I strongly suspect as a lot of crossover for AI, and at least has very similar mathematical structure.
e.g. optimization of state space control coefficients looks something like training a LLM matrix...
Re: Who Invented Backpropagation?
#26> BP's modern version (also called the reverse mode of automatic differentiation) So... Automatic integration? Proportional, integrative, derivative. A PID loop sure sounds like what they're talking about.
Reverse move automatic differentiation is not integration. It's still differentiation, but just a different method of calculating the derivative than the one you'd think to do by hand. It basically just applies the chain rule in the opposite order from what is intuitive to people. It has a lot more overhead than regular forwards mode autodiff because you need to cache values from running the function and refer back t…
I don't think most people think to do either direction by hand; it's all just matrix multiplication, you can multiply them in whatever order makes it easier.
Re: Who Invented Backpropagation?
#27When I worked on neural networks, I was taught David Rumelhart.
Re: Who Invented Backpropagation?
#28I have a question that's bothered me for quite a while now. In 2018, Michael Jordan (UC Berkeley) wrote a rather interesting essay - https://medium.com/@mijordan3/artificial-intelligence-the-re... (Artificial Intelligence — The Revolution Hasn’t Happened Yet) In it, he stated the following: > Indeed, the famous “backpropagation” algorithm that was rediscovered by David Rumelhart in the early 1980s, and which is now v…
> ... first arose in the field of control theory in the 1950s and 1960s. One of its early applications was to optimize the thrusts of the Apollo spaceships as they headed towards the moon.
I think "its" refers to control theory, not backpropagation.
Re: Who Invented Backpropagation?
#29Whatever the facts, the OP comes across as sour grapes. The author, Jürgen Schmidhuber, believes Hopfield and Hinton did not deserve their Nobel Prize in Physics, and that Hinton, Bengio, and LeCun did not deserve their Turing Award. Evidently, many other scientists disagree, because both awards were granted in consultation with the scientific community. Schmidhuber's own work was, in fact, cited by the Nobel Prize committee as background information for the 2024 Nobel.[a] Only future generations of scientists, looking at the past more objectively, will be able to settle these disputes.
[a] https://www.nobelprize.org/uploads/2024/11/advanced-physicsp...