Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

301–310 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#301

Post by CuriouslyC »

tfehring wrote: post #209
brucethemoose2 wrote: post #53

Earlier quoted context omitted.

Better data is still critical, even if bigger data isn't. The linked article emphasizes this.

I'd bet on a 2030 model trained on the same dataset as GPT-4 over GPT-4 trained with perfect-quality data, hands down. If data quality were that critical, practitioners could ignore the Internet and just train on books and scientific papers and only sacrifice <1 order of magnitude of data volume. Granted, that's not a negligible amount of training data to give up, but it places a relatively tight upper bound on the p…

It's possible that this effect washes out as data increases, but researchers have shown that for smaller data set sizes average quality has a large impact on model output.

Re: OpenAI’s CEO says the age of giant AI models is already over

#302

Post by ilaksh »

joebiden2 wrote: post #92

We will need a combination of technologies we have in order to really achieve emergent intelligence. Humans are comprised of various "subnets" modelling aspects which, in unison, produce self-conciousness and real intelligence. What is missing in the current line of approaches is that we only rely on auto-alignment of subnetworks by machine learning, which scales only up to a point. If we would produce a model which…

Check out David Shapiro's cognitive architecture stuff.

Re: OpenAI’s CEO says the age of giant AI models is already over

#303

Post by neom »

Andrex wrote: post #289
beowulfey wrote: post #211

Earlier quoted context omitted.

He’s not presenting false evidence here, he’s presenting a hunch. It’s a guess. No one is going to gain anything from this one way or another.

> No one is going to gain anything from this one way or another. Then why did Altman even bother to make this very public statement?

CEOs often make public statements for recruiting and to steer the company internally, it's important to mimmic what you say inside the company outside the company, maybe even more so.

Re: OpenAI’s CEO says the age of giant AI models is already over

#304

Post by ilaksh »

babyshake wrote: post #111
joebiden2 wrote: post #92

We will need a combination of technologies we have in order to really achieve emergent intelligence. Humans are comprised of various "subnets" modelling aspects which, in unison, produce self-conciousness and real intelligence. What is missing in the current line of approaches is that we only rely on auto-alignment of subnetworks by machine learning, which scales only up to a point. If we would produce a model which…

The other elements that may be required could be some version of the continuous sensory input that to us creates the sensation of "living" and, this one is a bit more philosophical, the sensation of suffering and a baseline establishment that the goal of the entity is to take actions that help it avoid suffering.

That's when it gets dangerous, when we try to really recreate animal (human) characteristics in digital form. Combining that with likely 1000-1000000 X increases in performance leads to these superintelligent digital creatures taking over.

Instead, we can focus on the Star Trek computer type stuff that we have with GPT and be incredibly careful about deploying those more animal/humanlike models and higher performance compute. Especially if we deliberately create the next species in digital form, make it 100X or 10000X faster thinking/smarter than us, and then enslave it, that is not only totally stupid but also proven unnecessary by the generality the latest AI models.

Re: OpenAI’s CEO says the age of giant AI models is already over

#305

Post by kccqzy »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?

Re: OpenAI’s CEO says the age of giant AI models is already over

#306

Post by rootw0rm »

brookst wrote: post #93
boringuser2 wrote: post #78

Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.

In this case I think it's Wired that's lying. Altman didn't say large models have no value, or that there will be no more large models, or that people shouldn't invest in large models. He said that we are at the end of the era where capability improvements come primarily from making models bigger. Which stands to reason... I don't think anyone expect us to hit 100T parameters or anything.

"640KB of memory ought to be enough for anybody!"

- Abraham Lincoln

Re: OpenAI’s CEO says the age of giant AI models is already over

#307

Post by smeyer »

kccqzy wrote: post #305
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?

No. I don't always assume that just because a CEO makes a public statement they have internal data backing up the claim. Sometimes they do! Other times, they have data but are misinterpreting it or missing something, but it's impossible to tell if the data is just internal. Other times they're making a statement without data based on their personal beliefs. Other times, they don't even think the statement is true but are saying it for messaging, marketing, or communication reasons!

Like the previous commenter, I'd be much more confident an asymptote was reached if it was being demonstrated publicly.

Re: OpenAI’s CEO says the age of giant AI models is already over

#308

Post by wetpaws »

kccqzy wrote: post #305
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?

He might or might not, this is a point of the claim.

Re: OpenAI’s CEO says the age of giant AI models is already over

#309

Post by ilaksh »

iandanforth wrote: post #184

There's plenty of room for models to continue to grow once efficiency is improved. The basic premise of the Google ML pathways project is sound, you don't have to use all the model all the time. By moving to sparse activations or sparse architectures you can do a lot more with the same compute. The effective model size might be 10x or 100x GPT-4 (speculated at 1T params) but require comparable or less compute. While…

The efficiency thing is what worries me. I think Nvidia has rough ideas for increasing efficiency 100 - 1000 times without changing the fundamental paradigm (i.e. memristors or crossbar arrays or something). If it doesn't go to 1000, I assume there is a lot of investment ramping up for realizing the new fully compute-in-memory systems.

If the software can leverage these efficiency gains effectively, then the concerns about runaway AI will be very relevant. Especially since people seem to think that they need to emulate all animal (like human) characteristics to get "real" general intelligence. Despite the fact that GPT is clearly general purpose. And people make no real differentiation between the most dangerous types of characteristics like self-preservation or full autonomy.

GPT shows that we can have something like a Star Trek computer without creating Data. People should really stop rushing their plans to create an army of Datas and then enslave them. Totally unnecessary and stupid.

Re: OpenAI’s CEO says the age of giant AI models is already over

#310

Post by ramshanker »

Wait till some one pulls a Cerberus for MEMORY. And than we connect a wafer scale processor with wafer scale memory and BOOM. New Mega size works emerge.

As for current claim, it might have to do with the amount of time spent taming the wildness of answer by raw GPT-4. So focus is shifting from increasing the model size.

Post reply on HN