51–60 of 120 posts
The chain rule was explored by Gottfried Wilhelm Leibniz and Isaac Newton in the 17th century. Either of them would have ”invented” backpropagation in an instant. It’s obvious.
Funny enough. For me it was the other way around. I always knew how to compute the chain rule. But really only understood what the chain rule means when I read up on what back propagation was.
Funny that hinton is not mentioned. Like how childish can the author be?
[HIN] J. Schmidhuber (AI Blog, 2020). Critique of Honda Prize for Dr. Hinton. Science must not allow corporate PR to distort the academic record.
Earlier quoted context omitted.
For what it's worth, it's a very mainstream opinion in the physics community that Hinton did not at all deserve a nobel prize in physics for his work. But that's because his work, and wasnt impactful at all to the physics community
I think Hinton himself has made that observation. In a recent talk he made a quip that he had to change some slides because if you have a Nobel prize in physics you should at least get the units right.
Who didn't? Depending on exactly how you interpret the notion of "inventing backpropagation" it's been invented, forgotten, re-invented, forgotten again, re-re-invented, etc, about 7 or 8 times. And no, I don't have specific citations in front of me, but I will say that a lot of interesting bits about the history of the development of neural networks (including backpropagation) can be found in the book Talking Nets:…
Once people had a sufficiently compelling reason to write differentiable code, the frameworks around differentiable programming (theano, tensorflow, torch, JAX) picked up a lot of steam.
Earlier quoted context omitted.
I think Hinton himself has made that observation. In a recent talk he made a quip that he had to change some slides because if you have a Nobel prize in physics you should at least get the units right.
Honest person would have rejected it and protected the prize's honour
Whatever the facts, the OP comes across as sour grapes . The author, Jürgen Schmidhuber, believes Hopfield and Hinton did not deserve their Nobel Prize in Physics, and that Hinton, Bengio, and LeCun did not deserve their Turing Award. Evidently, many other scientists disagree, because both awards were granted in consultation with the scientific community. Schmidhuber's own work was, in fact, cited by the Nobel Prize…
For what it's worth, it's a very mainstream opinion in the physics community that Hinton did not at all deserve a nobel prize in physics for his work. But that's because his work, and wasnt impactful at all to the physics community
Earlier quoted context omitted.
I think Hinton himself has made that observation. In a recent talk he made a quip that he had to change some slides because if you have a Nobel prize in physics you should at least get the units right.
Honest person would have rejected it and protected the prize's honour
Earlier quoted context omitted.
They're probably talking about Kalman Filters (1961) and LMS filters (1960).
To be fair, any multivariable regulator or filter (estimator) that has a quadratic component (LQR/LQE) will naturally yield a solution similar to backpropagation when an iterative algorithm is used to optimize its cost or error function through a differentiable tangent space.
Earlier quoted context omitted.
Funny enough. For me it was the other way around. I always knew how to compute the chain rule. But really only understood what the chain rule means when I read up on what back propagation was.
You don't think the people who invented the chain rule understood what it means?
I remember when I learnt about artificial neural networks at university in the late 00s my professors were really sceptical of them, rightly explaining that they become hard to train as you added more hidden layers.
See, what makes backpropagation and artificial neural networks work are all of the small optimisations and algorithm improvements that were added on top of backpropagation. Without these improvements it's too computationally inefficient to be practical and you have to contend with issues like exploding gradients.
I think Geoffrey Hinton has noted a few times that for people like him who have been working on artificial neural networks for years it's quite surprising that today neural networks just work because for years it was so hard to get them to do anything. In this sense while backpropagation is the foundational algorithm, it's not sufficient on it's own. It was the many improvements that were made on top of backpropagation that actually make artificial neural networks work and take off in the 2010s when some of the core components of modern neural networks started to fall into place.
I remember when I first learnt about neural networks I thought maybe coupling them with some kind of evolutionary approach might be what was needed to make them work. I had absolutely no idea what I was doing of course, but I spent so many nights experimenting with neural networks. I just loved the idea of an artificial "neural network" being able to learn a new problem and spit out an answer. The biggest regret of my life was coming out of university and going into web development because there were basically no AI jobs back then, and no such thing as an AI startup. If you wanted to do AI back then you basically had to be a researcher which didn't interest me at the time.