Who invented deep residual learning?
people.idsia.ch
Who invented deep residual learning?
1–10 of 37 posts
Re: Who invented deep residual learning?
#2I thought it was ResNet that invented the technique, but it's interesting to see it rooted back through LSTM which feels like a very architecture. ResNet really made massive waves in the field, and it was hard finding a paper that didn't reference it for a while.
Re: Who invented deep residual learning?
#3From the domain, I'm guessing the answer is Schmidhuber.
Re: Who invented deep residual learning?
#4Of all Schmidhuber's credit-attribution grievances, this is the one I am most sympathetic to. I think if he spent less time remarking on how other people didn't actually invent things (e.g. Hinton and backprop, LeCun and CNNs, etc.) or making tenuous arguments about how modern techniques are really just instances of some idea he briefly explored decades ago (GANs, attention), and instead just focused on how this single line of research (namely, gradient flow and training dynamics in deep neural networks) laid the foundation for modern deep learning, he'd have a much better reputation and probably a Turing award. That said, I do respect the extent to which he continues his credit-attribution crusade even to his own reputational detriment.
Re: Who invented deep residual learning?
#5I spent some time in the academia.
The person with whom an idea ends up associated often isn't the first person to have the idea. Most often is the person who explains why the idea is important, or find a killer application for the idea, or otherwise popularizes the idea.
That said, you can open what Schmidhuber would say is the paper which invented residual NNs. Try and see if you notice anything about the paper that perhaps would hinder the adoption of its ideas [1].
[1] https://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdv...
Re: Who invented deep residual learning?
#6The notion of inventing or creating something in ML doesn't seem very important as many people can independently come up with the same idea. Conversely, you can create novel results just by reviewing old literature and demonstrating it in a project.
Re: Who invented deep residual learning?
#7I spent some time in the academia. The person with whom an idea ends up associated often isn't the first person to have the idea. Most often is the person who explains why the idea is important, or find a killer application for the idea, or otherwise popularizes the idea. That said, you can open what Schmidhuber would say is the paper which invented residual NNs. Try and see if you notice anything about the paper tha…
Surely they wrote some papers in English even if they wrote their dissertation in German? Most people don’t go straight to dissertations anyway, it’s more of a place to go after you read a much shorter paper.
Re: Who invented deep residual learning?
#8I spent some time in the academia. The person with whom an idea ends up associated often isn't the first person to have the idea. Most often is the person who explains why the idea is important, or find a killer application for the idea, or otherwise popularizes the idea. That said, you can open what Schmidhuber would say is the paper which invented residual NNs. Try and see if you notice anything about the paper tha…
Surely they wrote some papers in English even if they wrote their dissertation in German? Most people don’t go straight to dissertations anyway, it’s more of a place to go after you read a much shorter paper.
Correct, that's [2]. In [2] they even say "[we] derive de main result using the approach first proposed in " and cite [1]. So the paper that everyone knows, in English (and with Bengio), explictly say that the original idea is in a paper in German, and still the scientific community chose not to cite the German original.
[1] https://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdv...
Re: Who invented deep residual learning?
#9The notion of inventing or creating something in ML doesn't seem very important as many people can independently come up with the same idea. Conversely, you can create novel results just by reviewing old literature and demonstrating it in a project.
That's how all/most science normally works.
Conversely, a huge amount of science is just scientists going "here's something I found interesting" but no one can figure out what to do with it. Then 30 or 100 years go by and it's a useful in a field that didn't even exist at the time.
Re: Who invented deep residual learning?
#10To comment on the substance.
It seems that these two people Schimidhuber and Hochreiter were perhaps solving the right problem for the wrong reasons. They thought this was important because they expected that RNNs could hold memory indefinitely. Because of BPTT, you can think of that as a NN with infinitely many layers. At the time I believe nobody worries about vanishing gradient for deep NNs, because the compute power for networks that deep just didn't exist. But nowadays that's exactly how their solution is applied.
That's science for you.