Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unpredictable abilities emerging from large AI models

quantamagazine.org

Post reply on HN
Print view View original

301–310 of 326 posts

Re: Unpredictable abilities emerging from large AI models

#301

Post by saiojd »

YeGoblynQueenne wrote: post #291
saiojd wrote: post #258

Earlier quoted context omitted.

I would not be so sure about compute capacity? Neural network architectures are still in their infancy, it is very likely that more efficient approaches exist.

We've had neural nets since 1943. The architectures are not "in their infancy", new architectures have been developing for decades, even entire neural net paradigms (feed-forward nets, recurrent nets, recursive nets, etc etc.). Their scale has also been increasing ever since Hinton and friends rediscovered backprop in the '80s. Neural nets are positively ancient at this point, not "in their infancy"! I don't know why…

> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes?

I don't have social media outside of HN.

It comes from a few observations:

1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high).

2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time.

3) https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times.

I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).

Re: Unpredictable abilities emerging from large AI models

#302

Post by saiojd »

datpiff wrote: post #282
saiojd wrote: post #258

Earlier quoted context omitted.

I would not be so sure about compute capacity? Neural network architectures are still in their infancy, it is very likely that more efficient approaches exist.

Sure, but the small & efficient networks so far have been pretty poor. Edge AI was hugely hyped a few years ago and it petered out. All of the big tech companies seems to be chasing the biggest networks with the largest training sets, that seems to be the direction for now. There could be huge efficiency savings within the implementations but when training costs are already this high it seems naive to think the low-h…

I wouldn't be so confident. Flash attention is a recent, significant improvement to training times and sure looks low-hanging now.

I was not familiar with Edge AI, interesting concept. I feel like improving the efficiency of very large models is much more likely.

The recent successes will lead to an even larger influx of $ in the short term --- this is an existential threat to Google, after all. We will see where things go!

Re: Unpredictable abilities emerging from large AI models

#303

Post by blurbleblurble »

YeGoblynQueenne wrote: post #294
blurbleblurble wrote: post #112

Earlier quoted context omitted.

I feel very frustrated with these takes because instead of grappling with what we're going to do about it (like having a conversation) it's a flat, dismissive denial, and it isn't even grounded in the science, which says that "memory augmented large language models are computationally universal". So at the very least we're dealing with algorithms that can do anything a hand written program can do, except that they've…

>> I feel very frustrated with these takes because instead of grappling with what we're going to do about it (like having a conversation) it's a flat, dismissive denial, and it isn't even grounded in the science, which says that "memory augmented large language models are computationally universal" That's not what "the science says", it's the title of an article that someone put on arxiv. The article has no theoretic…

That's a good paper! Thanks for sharing.

Re: Unpredictable abilities emerging from large AI models

#304

Post by lamp987 »

coldtea wrote: post #209
lamp987 wrote: post #135

Earlier quoted context omitted.

"your comment confidently states this is unfixable - presumably based on the frequency you've seen similar text on the internet. why should anyone believe the veracity of your statement? " no its because GPT is based on transformers.

and you aren't? Aren't you just a function of your input and memories (stuff you've read, sensory input) as run through/managed by some neural network? What makes you think the rest isn't just emergent properties? And what makes you think you can't hook up the LLM with some algorithms or layers that handle some of the rest behavior of what your brain does?

no, our brains arent based on transformers.

and the issue of lost uncertainty is inherrent to this, yes.

to fix this, a new type of llm would have to be invented. this particular branch of development may very well be a dead end.

Re: Unpredictable abilities emerging from large AI models

#305

Post by YeGoblynQueenne »

saiojd wrote: post #301
YeGoblynQueenne wrote: post #291

Earlier quoted context omitted.

We've had neural nets since 1943. The architectures are not "in their infancy", new architectures have been developing for decades, even entire neural net paradigms (feed-forward nets, recurrent nets, recursive nets, etc etc.). Their scale has also been increasing ever since Hinton and friends rediscovered backprop in the '80s. Neural nets are positively ancient at this point, not "in their infancy"! I don't know why…

> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes? I don't have social media outside of HN. It comes from a few observations: 1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could a…

>> It comes from a few observations:

Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here?

>> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist".

Half of your "claim" is incorrect. Don't just double down on it! There's really nothing to "claim" here, the "infancy" or not of neural nets is not a matter of claiming or guessing. Either you know what it is, or you don't.

Re: Unpredictable abilities emerging from large AI models

#306

Post by Imnimo »

pedrovhb wrote: post #10

This caught my attention as I found it implausible: > One DeepMind engineer even reported being able to convince ChatGPT that it was a Linux terminal and getting it to run some simple mathematical code to compute the first 10 prime numbers. Remarkably, it could finish the task faster than the same code running on a real Linux machine. Following the link, there's a screenshot to a screenshot [0] of a code-golf solutio…

Yeah, I tried asking GPT-4 to write a simple Python program to find the millionth prime. It wrote the naive approach where you loop over numbers and for each number, check all of the possible divisors up to its square root, and then keep track of how many primes you've found. Obviously, this takes quite a while to execute for N = a million. And yet, if you ask GPT-4 what the output of the code will be, as if my magic (memorization) it immediately spits out 15485863. It makes it a lot more clear that it's not actually simulating the code.

Re: Unpredictable abilities emerging from large AI models

#307

Post by saiojd »

YeGoblynQueenne wrote: post #305
saiojd wrote: post #301

Earlier quoted context omitted.

> Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes? I don't have social media outside of HN. It comes from a few observations: 1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could a…

>> It comes from a few observations: Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here? >> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". Half…

Your tone comes off as very adversarial/angry. Intentional or not? The claim is about the likelihood of more efficient approaches existing, not about the subjective qualifier preceding it (misunderstanding?). Yes, NNs have existed for a while now, but they will also exist for a long time forward in the future (hence the perhaps poor choice of word: infancy).

"if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future"

I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?

Re: Unpredictable abilities emerging from large AI models

#308

Post by muxator »

civilized wrote: post #214
coldtea wrote: post #206

Earlier quoted context omitted.

> What they said 10 years ago was correct. It did hit society like a sledge hammer. Machine learning basically took over the AI space and penetrated the consumer space with applications that were all but impossible in the previous decade. There's AI chips in smart phones now. And still almost all of these applications are not really impactful or that important compared to actually society and life changing developmen…

I feel like the only form of "AI" I regularly benefit from is when I type "n" in the address bar and it autocompletes to news.ycombinator.com. Oh, browser, you know me so well.

Yeah, I understand. But, really, there's no need for any AI in that.

Re: Unpredictable abilities emerging from large AI models

#309

Post by electrondood »

Animats wrote: post #245
guns wrote: post #91

Earlier quoted context omitted.

This is what Stephen Wolfram concludes in a recent article about ChatGPT: > The specific engineering of ChatGPT has made it quite compelling. But ultimately (at least until it can use outside tools) ChatGPT is “merely” pulling out some “coherent thread of text” from the “statistics of conventional wisdom” that it’s accumulated. But it’s amazing how human-like the results are. And as I’ve discussed, this suggests some…

> (at least until it can use outside tools) This is key. ChatGPT/GPT-4 alone are limited to reformulating what they know from their training data. Linked to search engines, databases, and computational tools such as Wolfram Alpha, they acquire much more capability. We're already seeing that with Microsoft Bing. (Update: what happens as large language models learn Excel? Especially since Microsoft is already connectin…

> Linked to search engines, databases, and computational tools such as Wolfram Alpha, they acquire much more capability.

I see this as analogous to the human brain; there are different structures which are particularly good at specific tasks/functions. They all work together.

The only difference between a human brain and an ANN is a difference of degree. A neuron and an artificial neuron are functionally identical. I think as we start interconnecting these models we see surprising emergent properties.

Re: Unpredictable abilities emerging from large AI models

#310

Post by YeGoblynQueenne »

saiojd wrote: post #307
YeGoblynQueenne wrote: post #305

Earlier quoted context omitted.

>> It comes from a few observations: Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here? >> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". Half…

Your tone comes off as very adversarial/angry. Intentional or not? The claim is about the likelihood of more efficient approaches existing, not about the subjective qualifier preceding it (misunderstanding?). Yes, NNs have existed for a while now, but they will also exist for a long time forward in the future (hence the perhaps poor choice of word: infancy). "if you want to know whether neural nets are in their 'infa…

My problem is that I read what people write on HN and treat it with the same seriousness and respect I want people to treat my comments, when the majority are only saying whatever comes to their mind just to make some sort of impression. Then when I point out some obvious error, people freak out and get defensive because they never expected anyone to take them seriously, they're just spouting off whatever without really thinking. And then they try to wiggle out of the conversation, just like you're doing right now, pretending that you were misunderstood.

Well, my mistake then for taking you seriously. Many apologies. You can rest assured it won't happen again.

Post reply on HN