Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

311–320 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#311

Post by zamnos »

kccqzy wrote: post #305
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

If OpenAI's CEO is making this claim, don't you think he has internal data backing up the claim?

Only OpenAI and its CEO know the full details on GPT-4's sizes so that's entirely possible. But since it's an internal secret, there's nothing compelling him to tell the truth. For all we know, he has internal data backing up the opposite of the claim but is making this claim so as to discourage potential competitors from spending the money training an even bigger and competitive ML model. Sending potential competitors off on a wide goose chase that, when pushed, he can just say "oh our internal data (that no one outside of a trusted few have seen) said otherwise". I have no idea if sama is such a person, but you must admit that the possibility exists.

Re: OpenAI’s CEO says the age of giant AI models is already over

#312

Post by MichaelZuo »

AlecSchueler wrote: post #239
mensetmanusman wrote: post #235

Earlier quoted context omitted.

Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.

Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.

Exponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query.

Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.

Re: OpenAI’s CEO says the age of giant AI models is already over

#314

Post by TaylorAlexander »

bhouston wrote: post #270
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training?

One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money.

That said I am curious if anyone else can really comment on this. It seems like as we get to very large and expensive models we will produce more and more specialized technology.

Re: OpenAI’s CEO says the age of giant AI models is already over

#315

Post by bhouston »

TaylorAlexander wrote: post #314
bhouston wrote: post #270

Earlier quoted context omitted.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to ver…

> They don't need to have the best chip tech available as long as they have enough money.

That sounds like someone who is "Blitzscaling." Costs do not matter in those cases, just acquiring customers and marketshare. But for the rest of us, who will see benefits but are not trying to win a $100B market, we will cost optimize.

Re: OpenAI’s CEO says the age of giant AI models is already over

#316

Post by kvetching »

boringuser2 wrote: post #78

Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.

Right. All the evidence points to more potential being left on the table for emergent abilities. It would make no sense that the model would develop all of these complex skills for better predicting the next token, then just stop.

It's a massive bet for a company to push compute into the billion dollar range - if saying something like this has the potential to help ward off those decisions, I don't see what's stopping them from saying it.

I basically see Microsoft talking when Sam talks.

Re: OpenAI’s CEO says the age of giant AI models is already over

#317

Post by kvetching »

olalonde wrote: post #199
cowmix wrote: post #133

Earlier quoted context omitted.

On the Lex Fridman podcast, he pretty much admitted he's not an AI (per se) and isn't the most excited about the tech (as he could be).

> he pretty much admitted he's not an AI Yeah, I also had a hunch he wasn't an AI. (I assume you meant "AI researcher" there :)) All joking aside, I wonder how that's affecting company morale or their ability to attract top researchers. I know if I was a top AI researcher, I'd probably rather work at a company where the CEO was an expert in the field (all else being equal).

I think Ilya leading the tech is plenty morale boosting

Re: OpenAI’s CEO says the age of giant AI models is already over

#318

Post by kvetching »

bla3 wrote: post #297

In other words, "this approach is getting to expensive for us and we need to control the narrative for our model sizes going down". Most increases in Imagenet etc scores came from bigger models. "The Unreasonable Effectiveness of Data" has aged very well. It seems very convenient for OpenAI that this trend should be over a few months after their "eye-wateringly expensive" comment.

They could have it open source and people like Gerganov could rewrite the code in C/C++, but god forbid they actual are open

Re: OpenAI’s CEO says the age of giant AI models is already over

#319

Post by epups »

MichaelZuo wrote: post #312
AlecSchueler wrote: post #239

Earlier quoted context omitted.

Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.

Exponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query. Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.

Compute will also scale exponentially in coming years. The data source limitation seems to be a harder barrier, I think many companies are experimenting with AI generated content for training at this point.
Post reply on HN