worth separating: LSTM (Hochreiter & Schmidhuber 1997) is ironclad and widely cited. the transformer attention priority claims are far shakier. conflating them is how Schmidhuber undermines himself
Munich 1991: The Roots of the Current AI Boom
51–60 of 108 posts
Re: Munich 1991: The Roots of the Current AI Boom
#52There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…
> From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc.
Well... that's "starve the beast" in action. A lot of things we take for granted, that underpin our modern ways of life, came to be due to government investing. Laser, radar, microwaves, the early Internet, that all was military R&D.
"Unfortunately" (well, for the rich and the MIC, at least) there is no way for people to siphon off money in government-funded research, so once the libertarian/small-state BS completely took over following the collapse of the USSR, a lot of that got torn down or supplemented with enough bureaucracy to make Germans cry... and that's why reusable rockets were not invented at NASA but at SpaceX instead.
Re: Munich 1991: The Roots of the Current AI Boom
#53Instead of focusing on the future, EU is busy rewriting history to please some eccentric researcher that claims he invented it all.
How does the EU feature in TFA exactly?
Re: Munich 1991: The Roots of the Current AI Boom
#54Earlier quoted context omitted.
That's what bugs me about him. So much work has gone into today's models that calling his contributions "the root" isn't really warranted. He's always complaining that Hinton, LeCun, and Bengio get more credit than they deserve, and now he's over-claiming himself.
Both can be right.
They could be, but they really aren't.
Name a single aspect of something modern like the Transformer architecture or how it is trained, that is even indirectly attributable to Schmidhuber.
No doubt he'd be jumping up and down wanting to take credit for residual connections, but where was Schmidhuber in the ImageNet era when everyone else was discovering how to build deep neural nets? Why didn't Schmidhuber invent ResNets, but instead waited until someone else (Kaiming He) did, then claim credit for it?
I'll bet Schmidhuber isn't done with yet ... when someone eventually comes up with an architecture for AGI, Schmidhuber will come out of the woodwork and point to a note he made on a napkin in 1800 that predicted it all.
Re: Munich 1991: The Roots of the Current AI Boom
#55Earlier quoted context omitted.
[flagged]
Hm, first time I see that Russian bots came to HN, but here we are. The history of comments of this account is insane.
I believe just someone who got too deep into Russian propaganda rabbid hole. The comment is too leaky for somebody working in St. Petersburg / a LLM.
Re: Munich 1991: The Roots of the Current AI Boom
#56The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's. More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA lib…
And Google's acquisition of DNN Research to get the ball rolling with conv nets and AI moneyball, followed by the acquisition of Deepmind. Schmidhuber IMO *has* been recognized as one of the 4 horseman and rightly so, but what has he done lately? Just noticed they now say the 3 godfathers of AI. This is what people hate about academia. It's not academia itself, it's the mean girl politics that emerge from the tenure system. And at this point, tenure should be abolished IMO having been utterly weaponized to defend the status quo.
Re: Munich 1991: The Roots of the Current AI Boom
#57There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…
Private labs feed off academia. Without academics to staff them, they'd get a lot less far.
I used to work at Nokia Research when they still made phones. Probably the closest thing Europe had to Silicon Valley twenty years ago. Except it was in Helsinki. Lots of stuff got invented there. Nokia didn't really manage to capitalize on its own inventions of course. Or rather it got caught up in its own clumsy attempts throwing babies out of the window by the bucket load. But others sure did. A lot of modern smart phones still have tech in them that Nokia pioneered before either Google or Apple shipped a smart phone.
At the time there was a lot of talk about the demise of industrial research labs. Bell labs (now actually owned by Nokia!), Xerox PARC, IBM, and all the other big US labs that produced amazing stuff are former shadows of themselves. There is some truth in that
But you could argue that Google and Apple picked up some of the slack. And the current AI boom came out of Google cherry picking all the best universities for their AI talent and putting them all together in a research group that then got free reign. Like Nokia, that involved a lot of ejecting of babies with the bath water. But it seems to have spawned lots of new startups that can trace their roots back to that research group in Google.
Re: Munich 1991: The Roots of the Current AI Boom
#58Instead of focusing on the future, EU is busy rewriting history to please some eccentric researcher that claims he invented it all.
How does the EU feature in TFA exactly?
There seems to be a coordinated push around Schmidhuber all around media in the EU, even LinkedIn is full of "random" posts about him in the past week.
Re: Munich 1991: The Roots of the Current AI Boom
#59Schmidhuber will NEVER stop trying to aggressively preserve his relevance and its endlessly amusing. Good for him.
Re: Munich 1991: The Roots of the Current AI Boom
#60Also see Schmidhuber's take on the Hinton + Hopfield Nobel prize: https://people.idsia.ch/~juergen/physics-nobel-2024-plagiari...
Read through his papers, and these are substantial accusations. Perhaps Hinton et al. should investigate, and either a) correct themselves by properly citing these Munich researchers; or b) proove they did not base their work (unlikely) on these papers;
And then, as a whole, this weighs in favor of European scholars and also should properly inform the funding of similar research in the EU.
Writing the last in the light of a month-and-a-half wait (to date) for EuroHPC to process their own form where we submitted a funding request by no less than University + Private Company already established in the area + 4 alumni, two PHDs and one postdoc. Zero response since.