Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

321–330 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#321

Post by mov_eax_ecx »

olalonde wrote: post #230
vorticalbox wrote: post #223

Earlier quoted context omitted.

I feel most CEOs are not top of their field but rather people who can take a vision and run with it.

It might be true in general; however, AI research laboratories are typically an exception, as they are often led by experienced AI researchers or scientists with extensive expertise in the field.

And that's why they have a hard time getting their stuff out there and getting the money they need. I mean, trying to run a business like a research lab is kind of flawed, you know? And you don't always want some Musk-like character messing around with the basics of the company

Re: OpenAI’s CEO says the age of giant AI models is already over

#322

Post by kvetching »

cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

people seem to have forgotten about the multi-modal GPT-4

There's a ton of potential left on the table. The question is if transformers have hit their limit with GPT-4 or not.

It's a pretty simple equation when you think about it this way and why Sam would say they have hit their limit. Sam is basically Microsoft and they want to retain their lead. Once Google learns to put their data to use correctly, it's almost guaranteed game over for OpenAI if they want it to be.

Re: OpenAI’s CEO says the age of giant AI models is already over

#323

Post by TaylorAlexander »

bhouston wrote: post #315
TaylorAlexander wrote: post #314

Earlier quoted context omitted.

> I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to ver…

> They don't need to have the best chip tech available as long as they have enough money. That sounds like someone who is "Blitzscaling." Costs do not matter in those cases, just acquiring customers and marketshare. But for the rest of us, who will see benefits but are not trying to win a $100B market, we will cost optimize.

Yes, agreed. I would like to run large models at home without serious expense.

Re: OpenAI’s CEO says the age of giant AI models is already over

#324

Post by kvetching »

unionpivo wrote: post #168
HDThoreaun wrote: post #126

Earlier quoted context omitted.

Can google stop them? It’s trivial to download YouTube videos

It’s trivial to download some YouTube videos. But I am quite sure that if you start doing it at scale, google will notice. You could be sneaky, but people in this business talk (since they know another good paying job is just around the corner) so It would likely come out.

OpenAI can afford to build a scraping network, built with thousands of proxies.

Re: OpenAI’s CEO says the age of giant AI models is already over

#325

Post by blibble »

bhouston wrote: post #270
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> A transformer trained on all text and all of YouTube will be killer amazing at so much.

ah yes, a bot where the answer to everything is to buy ridge wallets and play raid shadow legends

Re: OpenAI’s CEO says the age of giant AI models is already over

#326

Post by sdenton4 »

habitue wrote: post #272
sdenton4 wrote: post #242

Earlier quoted context omitted.

Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?

> Why wouldn't we have already seen spiking networks contribute something by now? This presupposes we've explored this space thoroughly, and we haven't. When everything you do with NNs improves results (how it mostly is now), that means not enough people are trying out ideas and new things. I don't think you can invoke EMH-like reasoning quite yet. Give us a nice long winter like physics has had, and then we can use…

It's not strictly a question of trying Everything and seeing what sticks.

There's a hypothesis in the parent comment - better handling of the time domain will lead to better modeling - which is actually fairly independent of architecture. So, there's going to be a number of possible ways to build better time modeling, ranging from tweaks to existing architecture to completely rebuilding Rome. So, if better time modeling really is a limitation, you don't need to rebuild Rome to find out.

In fact, I might argue that S4 layers already provide this improved time handling in the current world, and is proving very successful, which would again widen the moat for SNNs.

Re: OpenAI’s CEO says the age of giant AI models is already over

#327

Post by daydream »

TaylorAlexander wrote: post #314
bhouston wrote: post #270

Earlier quoted context omitted.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to ver…

Whether or not cost matters much depends on your perspective.

If you’re OpenAI and GPT4 is just a step on the way to AGI, and you can amortize that huge cost over the hundreds of millions in revenue you’re gonna pull in from subscriptions and API use… then sure you’re probably not very cost sensitive. It could be 20% cheaper or 50% more expensive, whatever, it’s so good your customers will use it at a wide range of costs. And you have truckloads of money from Microsoft anyways.

If you’re a company or a developer trying to build a feature, whole new product, or an entire company on top of GPT then that cost matters a whole lot. The difference between $0.06 and $0.006 per turn could be infeasible vs. shippable.

If you’re trying to compete with OpenAI then you’re probably doing everything possible to reduce that training cost.

So, whether or not it matters - it really depends.

Re: OpenAI’s CEO says the age of giant AI models is already over

#328

Post by throwaway2037 »

nomel wrote: post #194
mnky9800n wrote: post #189

Earlier quoted context omitted.

or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…

> model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data…

Excellent points in your post. You wrote:

    There's an incredible amount of non-text data out there still. Especially technical data.
"Especially technical data." What does this part mean? Initially, I thought you meant things like images and video, but now I am confused.

Re: OpenAI’s CEO says the age of giant AI models is already over

#329

Post by swyx »

blibble wrote: post #325
bhouston wrote: post #270

Earlier quoted context omitted.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> A transformer trained on all text and all of YouTube will be killer amazing at so much. ah yes, a bot where the answer to everything is to buy ridge wallets and play raid shadow legends

inaccurate, how will I watch Netflix in foreign countries without Nord VPN?

Re: OpenAI’s CEO says the age of giant AI models is already over

#330

Post by TaylorAlexander »

daydream wrote: post #327
TaylorAlexander wrote: post #314

Earlier quoted context omitted.

> I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to ver…

Whether or not cost matters much depends on your perspective. If you’re OpenAI and GPT4 is just a step on the way to AGI, and you can amortize that huge cost over the hundreds of millions in revenue you’re gonna pull in from subscriptions and API use… then sure you’re probably not very cost sensitive. It could be 20% cheaper or 50% more expensive, whatever, it’s so good your customers will use it at a wide range of c…

Totally true.
Post reply on HN