Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Your Weapon: Survival Strategies for Depressed AI Academics

arxiv.org

Post reply on HN
Print view View original

11–20 of 195 posts

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#11

Post by visarga »

jimsimmons wrote: post #5
ChatGTP wrote: post #4

Earlier quoted context omitted.

So endless upside might not actually exist ?

It may or may not. I mean it’d be pretty hard to increase yields of GPU fabs or data center sizes another 100x. There are logistical limitations. Unless some Apollo level mission is created by a superpower, we will hit bottlenecks. Algorithmic innovation is the only long term bet.

I thought that too, a year ago. But then chatGPT Turbo came out (ten times cheaper), and a slew of 30B and 10B models that are decent. Now I believe we will be able to run non-trivial AI on trivial hardware. Not to mention the Stable Diffusion revolution, hardware requirements went down pretty fast. Even an old GPU from 5 years ago can generate images quickly. Some LLMs run on iPhones.

The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#12

Post by blackbear_ »

> the grant funding structure is such that it rewards safe and incremental research on popular topics [...] Therefore, universities should probably avoid making grant funding a condition for hires and promotions

There you go. Just fix this idiotic "publish (NeurIPS) or perish" attitude already!

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#13

Post by MrScruff »

Obviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks. Of course, any rational person might have expected this given how the one working example came into existence. In general it all looks like a pretty resounding endorsement of the David Chalmers position to me.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#14

Post by hakuseki »

MrScruff wrote: post #13

Obviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'm…

> Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks.

I would disagree with this description. An "emergent property of large networks" would be something that just appears when you wire together a large network.

To get intelligent behavior, it's not sufficient to wire together a large neural network. You also need to use an optimizer to train it on a large data set.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#15

Post by llamataboot »

hakuseki wrote: post #14
MrScruff wrote: post #13

Obviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'm…

> Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks. I would disagree with this description. An "emergent property of large networks" would be something that just appears when you wire together a large network. To get intelligent behavior, it's not sufficient t…

once upon a time the optimizer was die or reproduce...now it's a little faster and more convoluted...

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#16

Post by MrScruff »

hakuseki wrote: post #14
MrScruff wrote: post #13

Obviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'm…

> Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'merely' being an emergent property of large networks. I would disagree with this description. An "emergent property of large networks" would be something that just appears when you wire together a large network. To get intelligent behavior, it's not sufficient t…

Yes of course, I'm being loose in my language, but you know what I'm getting at.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#17

Post by numbers_guy »

jimsimmons wrote: post #3

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along. These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. And people w…

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway).

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#19

Post by mr_gibbins »

numbers_guy wrote: post #17
jimsimmons wrote: post #3

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along. These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. And people w…

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it…

As someone in this space I can attest that AI teaching in most (UK) universities is generally poor on detail, abstract and behind industry by at least 3-5 years.

Not to mention that there is zero appetite from undergrads or postgrads to get into the nitty-gritty of it. To learn CNNs at the deep-dive level you need calculus, at least differentiation and integration. Calculus or even pre-calculus doesn't form part of the degree programme for most compsci BScs any more, because it is 'too hard'.

The way most students 'learn' AI is to use a method out of a Python library with near-zero understanding of how it works, and regurgitate it for an assessment.

Professorial research staff in most UK universities are light-years from AI within industry, and there's no clear path to that gap tightening, especially while universities are being run like second-rate consulting houses (don't get me started on THAT).

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#20

Post by Tomis02 »

numbers_guy wrote: post #17
jimsimmons wrote: post #3

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along. These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time. And people w…

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it…

> Disregarding the fact that ~85% of machine learning academics do not have what it takes to be hired by OpenAI anyway

That's interesting, could you please elaborate?

Post reply on HN