Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Munich 1991: The Roots of the Current AI Boom

people.idsia.ch

Post reply on HN
Print view View original

41–50 of 108 posts

Re: Munich 1991: The Roots of the Current AI Boom

#41

Post by gom_jabbar »

jongjong wrote: post #35

It's crazy to think that if Elon Musk hadn't mentioned Schmidhuber, most people would have no idea. It's nauseating how all the researchers who happened to work for big tech got tons of media coverage but Schmidhuber and his team were getting zero coverage yet they made massive contributions. I bet there are many others not mentioned. Nobody even knows about Frank Rosenblatt. It's insane how distorted our perception…

Yes, Rosenblatt is another good example. I recently looked deeper into the development of the perceptron and it's absolutely fascinating.

Re: Munich 1991: The Roots of the Current AI Boom

#42

Post by jasonhong »

psb217 wrote: post #31
gillesjacobs wrote: post #22

Earlier quoted context omitted.

Of course, but if you haven't read them you also shouldn't cite them. And that's where Schmidhuber goes off the rails: publicly shaming published papers into citing you isn't good academic practice. It's bullying.

"if you haven't read them you also shouldn't cite them" -- this is wildly incorrect in an academic context. If I'm using ResNets, I should cite the original ResNet paper, even if I haven't read it. If I'm using Transformers, I should cite the original Transformer paper, even if I haven't read it. If my work is a direct extension of method B, and method B is a direct extension of method A, I should cite the source of…

It's worth pointing out that sometimes, some papers just become part of the general context of things and are no longer explicitly cited. Or people cite textbooks or general survey papers instead.

For example, take a look at Albert Einstein's Google Scholar profile. He's not the top cited physicist. Not even close. It's because other researchers don't explicitly cite his papers. https://scholar.google.com/citations?user=qc6CJjYAAAAJ&hl=en...

Same with Tim Berners-Lee and the World Wide Web. Imagine if his original paper were cited every time someone deployed a web site.

Re: Munich 1991: The Roots of the Current AI Boom

#44

Post by MrBuddyCasino »

eru wrote: post #33
MrBuddyCasino wrote: post #20

Earlier quoted context omitted.

This is a straw-man if I ever saw one. Practically no one is against hard science research, properly conducted. The issues are rampant fraud / p-hacking / unreproducible garbage mixed with an unhealthy dose of ideological monoculture and indoctrination, garnished with rising tuition prices while sitting on huge endowments in case of the Ivy Leagues.

> Practically no one is against hard science research, properly conducted. As long as you do that with your own money (or money got freely given from other people), sure. If you use taxpayer money, that's a different game.

There is a long list of grievances I have regarding the (mis-) use of taxpayer money, and funding the hard sciences is way, way down. I can’t even see it from where I stand.

Re: Munich 1991: The Roots of the Current AI Boom

#45

Post by candu »

elorant wrote: post #24
jcattle wrote: post #3

There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…

I do a lot of work that is based on academic research, aka building a proprietary sparse embedding model. My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers? This isn’t even touched so you’d then have to go and invent a bazillion of algorithms yourself to make your life ea…

Well, yeah. That's why we have "research & development" as a term.

What you're referring to is the "development" part of that. In some sense: the job you have _exists precisely because it's not part of the research phase_, and it's equally as valuable as the research part. Research is the proof of concept; development is scaling up and making production-ready and finding small efficiencies and so on.

From an industry perspective, it's tempting to conflate these, because that's what industry research labs are designed to do: integrated R&D. But that is not at all how academic research labs work.

Re: Munich 1991: The Roots of the Current AI Boom

#46

Post by gessha »

elorant wrote: post #24
jcattle wrote: post #3

There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…

I do a lot of work that is based on academic research, aka building a proprietary sparse embedding model. My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers? This isn’t even touched so you’d then have to go and invent a bazillion of algorithms yourself to make your life ea…

I jest but database design is its own sub field of computer science, maybe look into their papers?

Re: Munich 1991: The Roots of the Current AI Boom

#47

Post by HarHarVeryFunny »

The current AI boom has more to do with NVIDIA, and the popularity of computer gaming giving us GPU compute, than who was using neural networks back in 1990's.

More specifically, it was really AlexNet, the 2012 ImageNet entry, running on two NVIDIA GTX 580's, that highlighted the practicality and utility of running large scale neural nets on affordable hardware. CUDA had been released in 2006, but cuDNN (the CUDA library for neural nets) didn't come out until 2014 - after AlexNet had already kickstarted the demand.

What followed from AlexNet was a few years of intense competition on the ImageNet benchmark, and larger and larger/deeper neural nets (CNNs), which gave rise to a lot of the algorithms and concepts still used today such as residual connections (originally from ResNet), ADAM (training algorithm), ReLU/etc, normalization, dropout, etc... all the fundamentals that made building large neural nets possible.

Schmidhuber's continual reminding everyone that he was working on neural nets back in the 1990s is beyond tiresome. Yes, he should have been recognized alongside Hinton/Bengio/LeCun as one of the pioneers, but time for him to get over it.

Re: Munich 1991: The Roots of the Current AI Boom

#48

Post by fedeb95 »

jcattle wrote: post #3

There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…

I think most people forget the graph-like nature of scientific research. You don't have n useful papers and m useless ones by themselves, you have an interconnection of those. There may be isolated cliques of uselessness, but there isn't a clear correlation between academia and private research.

Many ideas come from philosophy, which many find useless.

Heraclitus discovered change back in ancient Greek, I don't know where we would be in scientific research without that (deliberately ignoring the debate about the originality of what we know about Heraclitus work). I bet his contemporaries found his "research" useless.

Re: Munich 1991: The Roots of the Current AI Boom

#49

Post by FrustratedMonky »

jcattle wrote: post #3

There's this crowd on HN which is very vocal against academia. From what I've seen, the main points are that academia isn't efficient, most of the science coming out of academia is useless and that the whole system is just a waste of taxpayers money. Instead, what is often argued, all good research is done in private labs. Then pointing to SpaceX, Moderna, OpenAI, Google, etc. And while it is very true that often the…

It's like the old saying "only 10% of my marketing budget is making a difference, I just don't know which 10%"

You don't know ahead of time, where the breakthrough will come from.

There is ton of research that sits on the shelf, and then years later, it gets re-combined with some other useless research, and boom, some big breakthrough.

This current attitude of all research is worthless, so it should be cancelled, is shooting our future selves in the face.

Re: Munich 1991: The Roots of the Current AI Boom

#50

Post by elorant »

gessha wrote: post #46
elorant wrote: post #24

Earlier quoted context omitted.

I do a lot of work that is based on academic research, aka building a proprietary sparse embedding model. My issue with academia is that they don’t bother to solve the practical issues. They tell you how to build a PPMI model, but what about hitting a database that’s 500TB to find co-occurrence numbers? This isn’t even touched so you’d then have to go and invent a bazillion of algorithms yourself to make your life ea…

I jest but database design is its own sub field of computer science, maybe look into their papers?

I did that too. Ending up building my own reverse index with a fixed-size vocabulary. But that's my issue, you start building one product and you end-up building ten in the process to solve all edge cases because no one bothered to research how things scale.
Post reply on HN