Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

181–190 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#181

Post by ericmcer »

sitkack wrote: post #88
thewataccount wrote: post #73

Earlier quoted context omitted.

I work for a company with the web development team. We have ~6 software developers. I'd love to be able to just have people submit their resume's and extract the data from there, but instead I'm going to build a form and make applicants fill it out because chatGPT is going to be at least $0.05USD depending on the length of the resume. I'd also love to have mini summeries of order returns summerized in human form, but…

5 cents. Per resume. $500 per 10k. 1-3 hours of a fully loaded engineers salary per year. You are being criminally cheap.

For a big company that is nothing but if you are bootstrapping and trying to acquire customers with an MVP racking up a $500 bill is frightening. What if you offer a free trial and blow up and end up with 5k+ bill.

Re: OpenAI’s CEO says the age of giant AI models is already over

#182

Post by famouswaffles »

quonn wrote: post #170
famouswaffles wrote: post #55

Earlier quoted context omitted.

No it doesn't. For one thing they're already at human performance. For another, i don't think you realize how expensive inference can get. Microsoft with no scant amount of available compute is struggling to run gpt-4 such that they're rationing it between subsidiaries while they try to jack up compute. So saying, it would be economically sound if it cost x10 or x100 what it costs now is a joke.

How are they at human performance? Almost everything GPT has read on the internet didn‘t even exist 200 years ago and was invented by humans. Heck, even most of the programming it does wasn‘t there 20 years ago. Not every programmer starting from scratch would be brilliant, but many were self taught with very limited resources in the 80s form example and discovered new things from there. GPT cannot do this and is ver…

>How are they at human performance?

Because it performs at least average human level (mostly well above average) on basically every task it's given.

"Invest something new" is a nonsensical benchmark for human level intelligence. The vast majority of people have never and will never invent anything new.

If your general intelligence test can't be passed by a good chunk of humanity then it's not a general intelligence test unless you want to say most people aren't generally intelligent.

Re: OpenAI’s CEO says the age of giant AI models is already over

#183

Post by roflyear »

spaceman_2020 wrote: post #90
thunderbird120 wrote: post #27

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…

Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…

My IQ is 140 and I'm far from exceptional.

Re: OpenAI’s CEO says the age of giant AI models is already over

#184

Post by iandanforth »

There's plenty of room for models to continue to grow once efficiency is improved. The basic premise of the Google ML pathways project is sound, you don't have to use all the model all the time. By moving to sparse activations or sparse architectures you can do a lot more with the same compute. The effective model size might be 10x or 100x GPT-4 (speculated at 1T params) but require comparable or less compute.

While not a perfect analogy it's useful to remember that the human brain has far more "parameters", requires several orders of magnitude less energy to train and run, is highly sparse, and does a decent job at thinking.

Re: OpenAI’s CEO says the age of giant AI models is already over

#185

Post by midland_trucker »

PUSH_AX wrote: post #128
spaceman_2020 wrote: post #91

Earlier quoted context omitted.

If you were devious enough, you could be listening in on billions of phone conversations and messages and adding that to your data set. This also makes me doubt that NSA hasn't already cracked this problem. Or that China won't eventually beat current western models since it will likely have way more data collected from its citizenry.

I wonder what percentage of phone calls would add anything meaningful to models, I imagine that the nature of most phone calls are both highly personal and fairly boring.

That's a fair point. Not at all like training on Wikipedia in which nearly every sentence has novelty to it.

Then again it would give you data on every accent in the country, so the holy grail for modelling human speech.

Re: OpenAI’s CEO says the age of giant AI models is already over

#186

Post by roflyear »

taylorius wrote: post #169
dauertewigkeit wrote: post #138

Earlier quoted context omitted.

Are intelligent people that valuable? There's lots of them at every university working for peanuts. They don't seem to be that valued by society, honestly.

IQ isn't all that. Mine is 140+ and I'm just a somewhat well paid software engineer. It's TOO abstract a metric in my view - for sure it doesn't always translate into real world success.

Right were very much in the same boat. I'm good at pattern recognition I guess. I learn things quickly. What else? I don't have magic powers really. I still get headaches and eat junk food.

Re: OpenAI’s CEO says the age of giant AI models is already over

#187

Post by KhoomeiK »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.

Re: OpenAI’s CEO says the age of giant AI models is already over

#188

Post by ryneandal »

jackmott42 wrote: post #9
fergie wrote: post #3

The most comforting AI news I have read this year.

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild

Personally, I'm less worried about AI than I am about what people using these models can do to others. Misinformation/disinformation, more believable scams, stuff like that.

Re: OpenAI’s CEO says the age of giant AI models is already over

#189

Post by mnky9800n »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever.

actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so clearly demonstrated that small data is usually bad data and sophisticated models are sometimes solving problems will small data sets instead of problems with data.

of course this paper came out the year friedman published his gradient boosting paper, i think random forest also was only recently published then as well (i think there is a paper from 1996 about RF and briemans two cultures paper came out this year where he discusses RF i believe), and this is a decade before gpu based neural networks. So times are different now. But actually i think the big difference is these days i probably ask chatgpt to write the boiler plate code for a gradient boosted model that takes data out of a relational database instead of writing it myself.

Re: OpenAI’s CEO says the age of giant AI models is already over

#190

Post by summerlight »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

https://twitter.com/SmokeAwayyy/status/1646670920214536193

Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.

Post reply on HN