Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

201–210 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#201

Post by hackerlight »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

It's not a wild claim when you have empirically well-validated scaling laws which make this very prediction.

Re: OpenAI’s CEO says the age of giant AI models is already over

#202

Post by chubs »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…

Re: OpenAI’s CEO says the age of giant AI models is already over

#203

Post by matchagaucho »

I wonder how much the scarcity and cost of Nvidia GPUs is driving this message?

Nvidia is in a perfect "Arms Dealer" situation right now.

Wouldn't be surprised to see the next exponential leap in AI models trained on in-house proprietary GPU hardware architectures.

Re: OpenAI’s CEO says the age of giant AI models is already over

#204

Post by MattPalmer1086 »

MuffinFlavored wrote: post #77
MattPalmer1086 wrote: post #45

Earlier quoted context omitted.

Thing is, you can't trust what you find on stack overflow or other sources either. And searching, reading documentation and so on takes a lot of time too. I've personally been using it to explore using different libraries to produce charts. I managed to try out about 5 different libraries in a day with fairly advanced options for each using chatGPT. I might have spent a day in the past just trying one and not to the…

> Thing is, you can't trust what you find on stack overflow or other sources either. Eh. An outdated answer will be called out in the comments/downvoted/updated/edited more often than not, no?

Maybe, maybe not. I get useful results from it, but it doesnt always work. And it's usually not quite what I'm looking for, so then I have to go digging around to find out how to tweak it. It all takes time and you do not get a working solution out of the box most of the time.

Re: OpenAI’s CEO says the age of giant AI models is already over

#205

Post by jacobr1 »

uoaei wrote: post #109
no_wizard wrote: post #61

Earlier quoted context omitted.

All the LC grinding may come in handy after all! /s What algorithms specifically show the most results upon improvement? Going into this I thought the jump of improvements were really related more advanced automated tuning and result correction, in which it could be done at scale as it were allowing a small team of data scientists to tweak the models until desired results were being achieved. Are you saying instead,…

Traditional CS may have something to do with slightly improving the performance by allowing more training for the same compute, but it won't be an order of magnitude or more. The improvements to be gained will be found more in statistics than CS per se.

I'm not sure. Methods like Chinchilla and Quantization have been able to reduce compute by more than an order of magnitude. There might very well be a few more levels of optimizations within the same statistical paradigm.

Re: OpenAI’s CEO says the age of giant AI models is already over

#206

Post by quonn »

famouswaffles wrote: post #182
quonn wrote: post #170

Earlier quoted context omitted.

How are they at human performance? Almost everything GPT has read on the internet didn‘t even exist 200 years ago and was invented by humans. Heck, even most of the programming it does wasn‘t there 20 years ago. Not every programmer starting from scratch would be brilliant, but many were self taught with very limited resources in the 80s form example and discovered new things from there. GPT cannot do this and is ver…

>How are they at human performance? Because it performs at least average human level (mostly well above average) on basically every task it's given. "Invest something new" is a nonsensical benchmark for human level intelligence. The vast majority of people have never and will never invent anything new. If your general intelligence test can't be passed by a good chunk of humanity then it's not a general intelligence t…

Yeah these intelligence tests are not very good.

I would argue some programmers do in fact invent something new. Not all of them, but some. Perhaps 10%.

Second the point is not whether everyone is by profession an inventor but whether most people can be inventors. And to a degree they can be. I think you underestimate that by a large margin.

You can lock people in a room and give them a problem to solve and they will invent a lot if they have the time to do it. GPT will invent nothing right now. It‘s not there yet.

Re: OpenAI’s CEO says the age of giant AI models is already over

#207

Post by TheDudeMan »

matchagaucho wrote: post #203

I wonder how much the scarcity and cost of Nvidia GPUs is driving this message? Nvidia is in a perfect "Arms Dealer" situation right now. Wouldn't be surprised to see the next exponential leap in AI models trained on in-house proprietary GPU hardware architectures.

Google has been using TPUs for years and continuously improving the designs.

Re: OpenAI’s CEO says the age of giant AI models is already over

#208

Post by shagie »

g_delgado14 wrote: post #134
boringuser2 wrote: post #78

Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.

IIRC Altman has no financial stake in the success or failure of OpenAI to prevent these sorts of conflicts of interests between OpenAI and society as a whole

https://www.cnbc.com/2023/03/24/openai-ceo-sam-altman-didnt-... (https://news.ycombinator.com/item?id=35289044 - 24 days ago; 158 points, 209 comments)

> OpenAI’s ChatGPT unleashed an arms race among Silicon Valley companies and investors, sparking an A.I. investment craze that proved to be a boon for OpenAI’s investors and shareholding employees.

> But CEO and co-founder Sam Altman may not notch the kind of outsize payday that Silicon Valley founders have enjoyed in years past. Altman didn’t take an equity stake in the company when it added the for-profit OpenAI LP entity in 2019, Semafor reported Friday.

Re: OpenAI’s CEO says the age of giant AI models is already over

#209

Post by tfehring »

brucethemoose2 wrote: post #53
tfehring wrote: post #20

Related reading: https://dynomight.net/scaling/ In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third. Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise…

Better data is still critical, even if bigger data isn't. The linked article emphasizes this.

I'd bet on a 2030 model trained on the same dataset as GPT-4 over GPT-4 trained with perfect-quality data, hands down. If data quality were that critical, practitioners could ignore the Internet and just train on books and scientific papers and only sacrifice <1 order of magnitude of data volume. Granted, that's not a negligible amount of training data to give up, but it places a relatively tight upper bound on the potential gain from improving data quality.

Re: OpenAI’s CEO says the age of giant AI models is already over

#210

Post by frozenport »

chubs wrote: post #202
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…

No. Winter means people have lost interest in the research.

If anything successes in ChatGPT etc will be motivation for continued efforts.

Post reply on HN