Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

291–300 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#291

Post by esafak »

spaceman_2020 wrote: post #175
yunwal wrote: post #99

Earlier quoted context omitted.

But we're still nowhere near that, or even near surpassing the skill of an average person at a moderately complex information task, and GPT-4 supposedly took hundreds of millions to train. It also costs a decent amount more to run inference on it vs. 3.5. It probably makes sense to prove the concept that generative AI can be used for lots of real work before scaling that up by another order of magnitude for potential…

There are a lot of employees at large tech consultancies that don't really do anything that can't be automated away by even current models. Sprinkle in some more specific training and I can totally see entire divisions at IBM and Accenture and TCS being made redundant. The incentive structures are perversely aligned for this future - the CEO who manages to reduce headcount while increasing revenue is going to be very…

How is that perverse? That is the logical incentive. The perverse one is that middle managers rise by hiring people needlessly and building fiefdoms.

Re: OpenAI’s CEO says the age of giant AI models is already over

#292

Post by teruakohatu »

sanxiyn wrote: post #69
mullingitover wrote: post #51

Earlier quoted context omitted.

Quality over quantity. Just building a model with a gazillion parameters isn't indicative of quality, you could easily have garbage parameters with tons of overfitting. It's like megapixel counts in cameras: you might have 2000 gigapixels in your sensor, but that doesn't mean you're going to get great photos out of it if there are other shortcomings in the system.

What overfitting? If anything, LLMs suffer from underfitting, not overfitting. Normally, overfitting is characterized by increasing validation loss while training loss is decreasing, and solved by early stopping (stopping before that happens). Effectively, all LLMs are stopped early, so they don't suffer from overfitting at all.

I don't disagree with you, these models may be underfitted, but overfitting is not explicitly defined by val vs. training loss, but rather how closely its output matches training data.

If you trained a MLP model where the number of parameters exceeded the data, it would be able to memorize the data and return a zero loss on training data. The larger the models are, the greater chance it memorizes the data, rather than the latent variables or distribution of the data.

Early LLMs, GPT2 (circa 2019) for example was definitely overfitting. I would frequently copy and paste output and find a reddit comment with the exact words.

Re: OpenAI’s CEO says the age of giant AI models is already over

#293

Post by darkerside »

mensetmanusman wrote: post #235
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.

INSUFFICIENT DATA FOR MEANINGFUL ANSWER

https://users.ece.cmu.edu/~gamvrosi/thelastq.html

Re: OpenAI’s CEO says the age of giant AI models is already over

#294

Post by esafak »

gardenhedge wrote: post #95

Sam Altman and OpenAI must be pretty nervous. They have first mover advantage but they hold no hook or moat. Unless they can somehow keep their improvements ahead of the rest of the industry then they'll be lost among a crowd.

On the other hand, if Microsoft can change enough people's habits of using Google to search, they'll be happy as a clam.

Re: OpenAI’s CEO says the age of giant AI models is already over

#295

Post by astrange »

spaceman_2020 wrote: post #90
thunderbird120 wrote: post #27

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…

Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…

This only makes sense if you use "IQ" ignoring the actual definition of "IQ", in which case it's silly to use numbers in your post to make it look technical.

IQ 1. can't be compared against generations of IQ tests 2. supposedly doesn't test education (of course, it actually does) 3. isn't real.

Re: OpenAI’s CEO says the age of giant AI models is already over

#296

Post by mcbuilder »

Barrin92 wrote: post #248
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

But yet it can understand a json data schema from example and write javascript to interact with a library that I fed it and asked it to understand. Yes, I know its limitations, but it can also surprise me.

Re: OpenAI’s CEO says the age of giant AI models is already over

#297

Post by bla3 »

In other words, "this approach is getting to expensive for us and we need to control the narrative for our model sizes going down".

Most increases in Imagenet etc scores came from bigger models. "The Unreasonable Effectiveness of Data" has aged very well. It seems very convenient for OpenAI that this trend should be over a few months after their "eye-wateringly expensive" comment.

Re: OpenAI’s CEO says the age of giant AI models is already over

#298

Post by not2b »

thehumanmeat wrote: post #213
summerlight wrote: post #190

Earlier quoted context omitted.

https://twitter.com/SmokeAwayyy/status/1646670920214536193 Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.

That is one AI CEO out of 10,000. Just because OpenAI may not be interested in a larger model in the short term doesn't mean nobody else won't pursue it.

Others might pursue a smaller model that works as well as a larger model. If that can be done, whoever does it can very effectively compete on price/performance.

It seems that to assume otherwise (the only way to improve is to get bigger) is to assume that OpenAI already has found the optimal architecture. That seems unlikely.

Re: OpenAI’s CEO says the age of giant AI models is already over

#300

Post by not2b »

Andrex wrote: post #289
beowulfey wrote: post #211

Earlier quoted context omitted.

He’s not presenting false evidence here, he’s presenting a hunch. It’s a guess. No one is going to gain anything from this one way or another.

> No one is going to gain anything from this one way or another. Then why did Altman even bother to make this very public statement?

To answer the people asking when GPT5 is coming out, probably.
Post reply on HN