Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Munich 1991: The Roots of the Current AI Boom

people.idsia.ch

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Munich 1991: The Roots of the Current AI Boom

#91

Post by Windchaser »

MrBuddyCasino wrote: post #20
jcattle wrote: post #3

There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…

This is a straw-man if I ever saw one. Practically no one is against hard science research, properly conducted. The issues are rampant fraud / p-hacking / unreproducible garbage mixed with an unhealthy dose of ideological monoculture and indoctrination, garnished with rising tuition prices while sitting on huge endowments in case of the Ivy Leagues.

> Practically no one is against hard science research, properly conducted

Eh, I grew up conservative evangelical, and they were pretty much always going to have a problem with research in evolution and astronomy. Same goes for the fossil fuel industry w.r.t. climate science.

When the scientific evidence interferes with religious doctrine or industrial paycheck, then yeah, folks are still going to have a problem with hard science research.

Re: Munich 1991: The Roots of the Current AI Boom

#92

Post by Scroll_Swe »

HarHarVeryFunny wrote: post #47

The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's. More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA lib…

Thanks AI for destroying my hobby. :)

Re: Munich 1991: The Roots of the Current AI Boom

#93

Post by HarHarVeryFunny »

nextos wrote: post #87
HarHarVeryFunny wrote: post #47

The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's. More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA lib…

I agree. I also think it's about the hardware and, obviously, recognizing AD as the fundamental primitive. Particular architectures don't matter so much yet. It's quite possible that S3-Mamba or xLSTM could be used in lieu of transformers and we would still have LLMs.

No doubt some aspects of the Transformer architecture are fungible, but as Hochreiter is implicitly proving you can't just scale up an LSTM and get Transformer level performance out of it, which is why he has come up with this new xLSTM architecture to try to do better!

The short 2K Transformer context size that Hochreiter is using for xLSTM comparisons seems a bit suspect ... Of course the attraction of an RNN is that it has "infinite" context/memory, so it may be expected to outperform a short context Transformer, while at the same time context scalability is an issue for RNNs, even an LSTM. Has he just cherry picked the size at which the advantages of an xLSTM outweigh the disadvantages ?

Note that despite the table saying GPT-3, he isn't actually testing against GPT-3 (a 175B model), but rather a 400M GPT closer to GPT-1 in size. The only reason he's calling it "GPT-3" is because of the 2K context size.

Could a 1T param xLSTM one-shot a compiler or find a needle in a 1M token haystack? Does an induction-head-like AB => A'B' in-context learning primitive, or something functionally equivalent, emerge out of stacked xLSTM layers?

At the end of the day it's prediction power that matters, not specific architecture, but we've yet to see any other architecture that functionally competes with a large Transformer. It would be neat to see a significantly different one that did!

Re: Munich 1991: The Roots of the Current AI Boom

#95

Post by itkovian_ »

Contrarian view; I think he’s right. Many of these ideas it’s almost shocking how many you can find sketched out in his old papers. To the point where I think it’s very wise to read all his work to see what hasn’t showed up yet but likely will. Artificial curiosity for example.

Re: Munich 1991: The Roots of the Current AI Boom

#96

Post by auggierose »

HarHarVeryFunny wrote: post #47

The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's. More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA lib…

> Schmidhuber's continual reminding everyone that he was working on neural nets back in the 1990s is beyond tiresome. Yes, he should have been recognized alongside Hinton/Bengio/LeCun as one of the pioneers, but time for him to get over it.

Not getting a turing award / nobel prize for your life's work, when other's got it for the same thing, I certainly would not get over that. To a comment like that, I would just think a polite, fuck you.

Re: Munich 1991: The Roots of the Current AI Boom

#97

Post by gillesjacobs »

psb217 wrote: post #31
gillesjacobs wrote: post #22

Earlier quoted context omitted.

Of course, but if you haven't read them you also shouldn't cite them. And that's where Schmidhuber goes off the rails: publicly shaming published papers into citing you isn't good academic practice. It's bullying.

"if you haven't read them you also shouldn't cite them" -- this is wildly incorrect in an academic context. If I'm using ResNets, I should cite the original ResNet paper, even if I haven't read it. If I'm using Transformers, I should cite the original Transformer paper, even if I haven't read it. If my work is a direct extension of method B, and method B is a direct extension of method A, I should cite the source of…

> this is wildly incorrect in an academic context.

Worked for me in my academic career ¯\_(ツ)_/¯ Of course if you suspect a work is poignant, you should read it, then cite it.

Re: Munich 1991: The Roots of the Current AI Boom

#98

Post by nextos »

HarHarVeryFunny wrote: post #93
nextos wrote: post #87

Earlier quoted context omitted.

I agree. I also think it's about the hardware and, obviously, recognizing AD as the fundamental primitive. Particular architectures don't matter so much yet. It's quite possible that S3-Mamba or xLSTM could be used in lieu of transformers and we would still have LLMs.

No doubt some aspects of the Transformer architecture are fungible, but as Hochreiter is implicitly proving you can't just scale up an LSTM and get Transformer level performance out of it, which is why he has come up with this new xLSTM architecture to try to do better! The short 2K Transformer context size that Hochreiter is using for xLSTM comparisons seems a bit suspect ... Of course the attraction of an RNN is th…

I am not sure I agree we've yet to see any other architecture that competes with a large transformer. For example, in long-range tasks such as those related to genome prediction, state-space models (Mamba) exhibit SOTA performance. I also think it's hard to separate architectural advantages from maturity, given that transformers have received much more attention.

Re: Munich 1991: The Roots of the Current AI Boom

#99

Post by Sharlin »

trilogic wrote: post #83
Sharlin wrote: post #69

Earlier quoted context omitted.

Somehow "protecting companies" by keeping basic research, done openly at a university lab, from being "stolen"? What? It's like saying it's painful that the Web was invented in Europe and opened for everybody rather than being kept at CERN to protect European companies.

Right, is totally fine to create new inventions, but let others take credit and financial benefits. It is our duty to protect and get the benefits of European inventions, especially the ones financed with public tax. Open for everybody means benefits for everybody.

Honestly can't tell if sarcasm. Are you seriously proposing that all university research just stop and be replaced with some new proprietary system?
Post reply on HN