Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

161–170 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#161

Post by deepsquirrelnet »

generalizations wrote: post #75

I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run. Pretty sure Microsoft swapped out Bing for something a lot smaller in the last couple of weeks; Google hasn't even tried to implement a publicly available large model. And OpenAI still has usage caps on their GPT-4. I'd bet that they can still see improvement in performance with GPT-5, but that when they look a…

> I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run.

I think it's likely that they're out of training data to collect. So adding more parameters is no longer effective.

> most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant.

I tend to agree, and I think their pathway toward this will all come from continuing advances in fine tuning. Instruction tuning, RLHF, etc seem to be paying off much more than scaling. I bet that's where their investment is going to be turning.

Re: OpenAI’s CEO says the age of giant AI models is already over

#162

Post by fpgaminer »

cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

> Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models.

Or augment the dataset using RLHF, which provides an "infinite" dataset to train LLMs on. Limited by the capabilities of the scoring model which, of course, you can scale the scoring model infinitely so again the limit isn't dataset size but training compute.

Re: OpenAI’s CEO says the age of giant AI models is already over

#163

Post by LesZedCB »

the way i see it, the expensive part should be to train the models via simulated architectures in GPUs or TPUs or whatever.

but once they are trained, is there a way to encode the base models into hardware where inference costs are basically negligible? hopefully somebody is seeing if this is possible, using structurally encoded hardware to make inference costs basically nil/constant.

Re: OpenAI’s CEO says the age of giant AI models is already over

#164

Post by mstolpm »

transcriptase wrote: post #115
cowpig wrote: post #82

Earlier quoted context omitted.

OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.

I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models…

Why are you discussing OpenAI with ChatGPT? I’m honestly interested.

I would imagine that any answer of ChatGPT on that topic is either (a) „hallucinated“ and not based on any verifiable fact or (b) scripted in by OpenAI.

The same question pops up for me whenever someone asks ChatGPT about the internals and workings of ChatGPT. Am I missing something?

Re: OpenAI’s CEO says the age of giant AI models is already over

#165

Post by gowld »

asdfman123 wrote: post #159
spaceman_2020 wrote: post #90

Earlier quoted context omitted.

Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…

The reason we put everyone through school is we believe that it’s in society’s best interest to educate everyone to the peak of their abilities. It’s good for many different reasons. It would be much easier to identify gifted kids and only educate them, but I happen to agree that universal education is better.

It would be much easier to identify gifted kids and only educate them

Is it so easy?

Re: OpenAI’s CEO says the age of giant AI models is already over

#166

Post by Vrondi »

spaceman_2020 wrote: post #94
mrtksn wrote: post #81

Earlier quoted context omitted.

You can transcribe all spoken words everywhere and keep the model up to date? Keep indexing new data from chat messages, news articles, new academic work etc. The data is not finite.

What about all the siloed content kept inside corporate servers? You won't get normal GPT to train on it, of course, but IBM could build a "IBM-bot" that has all the GPT-4 dataset + all of IBM's internal data. That model might be very well tuned to solve IBM's internal problems.

Since Chat-GPT-4 is being integrated into the MS Office suite, this is an "in" to corporate silos. The MS cloud apps can see inside a great many of those silos.

Re: OpenAI’s CEO says the age of giant AI models is already over

#167

Post by dopidopHN »

transcriptase wrote: post #115
cowpig wrote: post #82

Earlier quoted context omitted.

OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.

I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models…

Without getting into morality.

It’s pretty easy to have chatGPT contradict itself, point it out and have the LLM respond « well, I’m just generating text, nobody said it had to be correct »

Re: OpenAI’s CEO says the age of giant AI models is already over

#168

Post by unionpivo »

HDThoreaun wrote: post #126
neel8986 wrote: post #71

Earlier quoted context omitted.

And why will google allow them to do that at scale?

Can google stop them? It’s trivial to download YouTube videos

It’s trivial to download some YouTube videos.

But I am quite sure that if you start doing it at scale, google will notice.

You could be sneaky, but people in this business talk (since they know another good paying job is just around the corner) so It would likely come out.

Re: OpenAI’s CEO says the age of giant AI models is already over

#169

Post by taylorius »

dauertewigkeit wrote: post #138
spaceman_2020 wrote: post #90

Earlier quoted context omitted.

Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…

Are intelligent people that valuable? There's lots of them at every university working for peanuts. They don't seem to be that valued by society, honestly.

IQ isn't all that. Mine is 140+ and I'm just a somewhat well paid software engineer. It's TOO abstract a metric in my view - for sure it doesn't always translate into real world success.

Re: OpenAI’s CEO says the age of giant AI models is already over

#170

Post by quonn »

famouswaffles wrote: post #55
hcks wrote: post #46

Earlier quoted context omitted.

Yes, but it also tells us that if Altman is honest here, then he doesn’t believe GPT-like models can scale to near level human performances (because even if the cost of compute was 10x or even 100x it would still be economically sound).

No it doesn't. For one thing they're already at human performance. For another, i don't think you realize how expensive inference can get. Microsoft with no scant amount of available compute is struggling to run gpt-4 such that they're rationing it between subsidiaries while they try to jack up compute. So saying, it would be economically sound if it cost x10 or x100 what it costs now is a joke.

How are they at human performance? Almost everything GPT has read on the internet didn‘t even exist 200 years ago and was invented by humans. Heck, even most of the programming it does wasn‘t there 20 years ago.

Not every programmer starting from scratch would be brilliant, but many were self taught with very limited resources in the 80s form example and discovered new things from there.

GPT cannot do this and is very far from being able to.

Post reply on HN