Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Your Weapon: Survival Strategies for Depressed AI Academics

arxiv.org

Post reply on HN
Print view View original

81–90 of 195 posts

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#81

Post by jrsdav »

caddemon wrote: post #38
disgruntledphd2 wrote: post #31

Earlier quoted context omitted.

It's a function of the people who hang out here. If you like reading and text in general, you are both more likely to comment here and additionally regard language as the most important part of sentience/consciousness. This leads to believing that GPT has solved many of these issues.

As someone very language-focused, it has actually done the opposite for me: challenged some of my beliefs about how language enables intelligence. It's obviously skilled at language use, but most of its outputs where it has to make any sort of arguments or inferences read as incredibly stupid to me (even the ones that "tricked" people early on, some of them tricked me but in a way where I was about to write a respons…

Are you familiar with Pinker’s research presented in book The Language Instinct?

> language enables intelligence

In his research, he argues that it’s the other way around; we have an innate intelligence for acquiring language.

To put it another way, language has made us more knowledgeable (because it’s a great mechanism for building and transferring context), but intelligence or aptitude for language was present before we developed languages.

Does this align with how your views were reshaped post LLMs?

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#82

Post by muskmusk »

> If we have learned one thing from deep learning, it is that scaling works. From the ImageNet [15] competitions and their various winners to ChatGPT, Gato [13], and most recently to GPT-4 [1], we have seen that more data and more compute yield quantitatively and often even qualitatively better results. (By the time you are reading this, that list of very recent AI milestones might very well be outdated.). Of course there are improvements to learning algorithms and network architectures as well, but these improvements are only really useful in the context of the massive scale of experiments. (Sutton talks about the “Bitter Pill”, referring to that simple methods that scale well always win the day when more compute becomes available [18].) A scale that is not achievable by academic researchers nowadays. As far as we can tell, the gap between the amount of compute available to ordinary researchers and the amount available to stay competitive is growing every year.

>This goes a long way to explain the resentment that many AI researchers in academia feel towards these companies. Healthy competition from your peers is one thing, but competition from someone that has so much resources that they can easily do things you could never, no matter how good your ideas are, is another thing.

This sounds like teenagers whining they can't all be popular. Science isn't a competition to be won so that you can get praise and attention. Science exists to discover things that then hopefully are useful to people. If someone that isn't you discovers something useful that isn't bad because then you can't discover it. It is good because that is a problem solved.

The resources requirements FOR SOME SPECIFIC PROBLEMS have gone up to a ridiculous degree, but there are plenty of problems left to solve. In fact a new one that is at least as important has been created: Replicate current results with less hardware/parameters/whatever.

The difference between having GPT4 as a slow and costly service that requires network calls vs having it locally with almost no cost will be a huge achievement. Stop sulking and get to work!

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#83

Post by agravier »

gitfan86 wrote: post #54
bo1024 wrote: post #52

As a theorist in AI, the premise is funny but also really sad. And yet it is a very pervasive premise in the field. OpenAI is not doing science. They are building a big shiny thing, showing it off, keeping it closed, and making money off of it. That is not part of the scientific process. It is as if, every time SpaceX launched a rocket a little bit higher, every aerospace department ooh'd and aah'd and threw up their…

OpenAI wasn't created to "Do Science". It was created to give humanity a better chance at navigating the singularity.

Ooooh and here I was, an ignoramus thinking OpenAI was mostly trying to find a way to milk the generative models cash cow. Silly, cynical me, I should have known better.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#84

Post by FrustratedMonky »

galaxyLogic wrote: post #67
FrustratedMonky wrote: post #47

Earlier quoted context omitted.

Maybe there is also some burden on other side to providing a definition of terms for the other party to argue against. I agree the burden of proof is on the AI side, but there is some burden on the deniers to provide some definitions if they wish to continue denying. AI Side: look here is my proof, Deniers: NO That isn't what I meant. I'd say GPT is already by its existence proof enough to start at least discussing c…

> burden on other side to providing a definition of terms The skeptics are really not making any claims are they? If not then there's no burden of proof on them, is there? Whoever says or implies that ChatBots are or are "becoming" "sentient" are the ones making a claim. They should prove it if they make or imply such a claim. Skeptics are rightly skeptical, until they see a proof with clear definitions of terms from…

Guess I'm saying that is done. The problem is there are a dozen definitions of "Sentient", and many people have already written proofs on how AI can be "Sentient", and there are just as many people have written proofs on how AI can never be "Sentient". It comes around to definitions and understanding of the definitions. To just toss out the "AI can be sentient" side because they have the burden, is just the same "It's so obvious I don't have to bother" judgment we've seen throughout history.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#85

Post by revolvingocelot »

gitfan86 wrote: post #54
bo1024 wrote: post #52

As a theorist in AI, the premise is funny but also really sad. And yet it is a very pervasive premise in the field. OpenAI is not doing science. They are building a big shiny thing, showing it off, keeping it closed, and making money off of it. That is not part of the scientific process. It is as if, every time SpaceX launched a rocket a little bit higher, every aerospace department ooh'd and aah'd and threw up their…

OpenAI wasn't created to "Do Science". It was created to give humanity a better chance at navigating the singularity.

OpenAI was created to trade (literally) on the "open" in their name. Any advancements in actual technology are likely secondary to the billion fucking dollar investment (which Altman has suggested might not be enough!). It's certain they aren't here to do "science", which IME involves showing your work to others.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#86

Post by CuriouslyC »

MrScruff wrote: post #13

Obviously AI academics are uniquely challenged by the 'scaling is (nearly) everything' reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we're actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there's a bitter sweet aspect to the slightly mundane reality of it 'm…

Scaling isn't nearly everything though. Scaling is low hanging fruit, but there are real constraints that limit it. For starters, we have finite data and increasing the size of our data set is going to get progressively more expensive (new data is going to be more niche as well). Beyond that, running these models on device is way more compelling than using a service, and the best current RLHF tuned Llama models that run on consumer hardware are close enough to ChatGPT to be quite useful.

The race isn't to build the biggest model, the race is to build the model that uses data the most efficiently and produces the best results when constrained to parameter counters that can fit on consumer hardware.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#87

Post by chasd00 »

MrScruff wrote: post #35
southernplaces7 wrote: post #24

Earlier quoted context omitted.

Nothing about current AI models profoundly challenges the unknowns of what makes consciousness work in any mystery-resolving way. People who confuse plainly programmed algorithmic systems like GPT 4 or Midjourney with the still unresolved issues of sentience as we know it so far are drinking far too much of the current batch of AI punch. Sadly, it's a sentiment I see all too often on HN, a site in which i'd assume mo…

I think the mystery of why we're not just philosophical zombies was never going to have some eureka moment, given that it's quite plain it's an emergent property. Was there a creature in history that wasn't sentient that gave birth to a creature that was? Clearly not. The reason recent results are significant in this discussion is because: - Larger networks are already displaying emergent behaviours that couldn't hav…

Are you saying that consciousness didn’t result from, survival of the fittest, evolutionary pressure but more of a byproduct of a larger brain (network)? Like it emerged because of a larger network but was not selected for through evolution. That makes sense to me. Humans have a grotesquely large brain compared to other animals.

Edit: I wonder if there are studies that have determined whether or not new born infants are self aware or does it take time for it to emerge after the brain has been trained on its environment for a while

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#88

Post by SuoDuanDao »

api wrote: post #70
hyperbovine wrote: post #58

Earlier quoted context omitted.

Conversely: if you’re a depressed AI researcher: come on over to science, the water’s fine! We have centuries worth of open problems that are not going to be solved by ChatGPT, and need more smart technical people than we can hire.

The problem is usually that science can't pay even a fraction of what you can make in industry right now, and meanwhile the good science jobs are often in high cost of living places.

I really think we need a UBI so that people who want to can afford to do science how many drug addicts less motivated to quit is a scientist less motivated to maximize clicks worth? It's gotta be more than zero.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#89

Post by TeMPOraL »

matthewdgreen wrote: post #40
numbers_guy wrote: post #17

Earlier quoted context omitted.

I don't think they are putting themselves on a pedestal. It's more that they made the choice to remain in academia, because they believed that were trading a higher salary for the academic freedom and the chance to work with really cool problems. And then it turns out, industry are the ones really working with the really cool problems. (Disregarding the fact that ~85% of machine learning academics do not have what it…

I think your response is very narrow in defining what a "really cool problem" is. If in the 1970s you imagined that the cool problem in ECE/CS was "building ever-more-powerful computer processors" then you basically got smoked by Intel and other industry labs. But there's a different way to look at it. Because industry went off and solved the now-"boring" problem of building ever-more-powerful computer processors, th…

The "cool problem" is cracking the mystery of intelligence, of consciousness. Building a general AI. Until recently, this was mostly an academic pursuit, and people expected it to take decades or more. But now, suddenly, it's highly likely the problem will be cracked within a decade, and it will be done by corporate R&D teams, by means of scaling up the transformer architectures.

So I get how they feel - one of the coolest problems ever instantly went from being something anyone could hope to contribute to, to something almost no one can. You can't match the compute to do the things OpenAI & Google & friends do. And if you happen to stumble on something related that can be explored without access to obscene amounts of capital, guess what, the corporate research teams will notice it, and they can do it better than you, and then they can the idea much further than you ever could.

Re: Choose Your Weapon: Survival Strategies for Depressed AI Academics

#90

Post by gitfan86 »

agravier wrote: post #83
gitfan86 wrote: post #54

Earlier quoted context omitted.

OpenAI wasn't created to "Do Science". It was created to give humanity a better chance at navigating the singularity.

Ooooh and here I was, an ignoramus thinking OpenAI was mostly trying to find a way to milk the generative models cash cow. Silly, cynical me, I should have known better.

Don't worry about it. VERY few people (Sama, Elon, Me) saw this coming and see what is coming next.
Post reply on HN