Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

381–390 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#381

Post by nmfisher »

TaylorAlexander wrote: post #314
bhouston wrote: post #270

Earlier quoted context omitted.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training? One of the OpenAI guys was talking about this. He said the specific technology does not matter, it is just a cost line item. They don't need to have the best chip tech available as long as they have enough money. That said I am curious if anyone else can really comment on this. It seems like as we get to ver…

Maybe it's just a line item to them, but it's pretty relevant to anyone operating with a less-than-gargantuan budget. If a superior/affordable chip is widely available, OpenAI's competitive advantage recedes rapidly because suddenly everyone else can do what they can. To some extent that's exactly what happened with DALL-E/StableDiffusion.

assuming it's not horizontally scalable, because otherwise they would just out-spend everyone else anyway like they've already done. That's a big "if", though.

Re: OpenAI’s CEO says the age of giant AI models is already over

#382

Post by PartiallyTyped »

KhoomeiK wrote: post #187
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.

There's only so much first order derivatives can do for you.

My bet is on sparsity, lottery tickets and symmetries.

Re: OpenAI’s CEO says the age of giant AI models is already over

#383

Post by spaceman_2020 »

blibble wrote: post #325
bhouston wrote: post #270

Earlier quoted context omitted.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so. I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand lo…

> A transformer trained on all text and all of YouTube will be killer amazing at so much. ah yes, a bot where the answer to everything is to buy ridge wallets and play raid shadow legends

Your AI overlords, brought to you by NordVPN and HelloFresh.

Re: OpenAI’s CEO says the age of giant AI models is already over

#384

Post by nmfisher »

drippingfist wrote: post #255
Barrin92 wrote: post #248

Earlier quoted context omitted.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

"The third word of this sentence is "the," and its third letter is "e." - GPT-4

Just for funsies:

Me: If a gloop is a glam, and a glam is not a flom, what is the fourth letter of the ninth word in this sentence?

GPT4: The ninth word in this sentence is "sentence." The fourth letter of this word is "t".

Re: OpenAI’s CEO says the age of giant AI models is already over

#385

Post by jpalomaki »

neel8986 wrote: post #71
machdiamonds wrote: post #41

Earlier quoted context omitted.

Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685 There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.

And why will google allow them to do that at scale?

One way would be to get people to let AI watch as they interact with computer (watch YouTube or perform other tasks). You might even be able to outsource some of the computing to the local system.

Re: OpenAI’s CEO says the age of giant AI models is already over

#386

Post by DesiLurker »

fergie wrote: post #3

The most comforting AI news I have read this year.

I agree, probably for a different reason. IMO the fact that Sam is saying this likely means that the LLMs are close to the upper knee of the S-Curve and after GPT5 they likely dont have many new fundamental ideas (additional SCurves) to throw at it. since they are ahead of the rest, it likely means we might be headed for an AI improvement pause for couple of years after GPT5.

Still good enough to upset the balance in search/ad market. Interesting times.

Re: OpenAI’s CEO says the age of giant AI models is already over

#387

Post by The_Colonel »

ericabiz wrote: post #215
chubs wrote: post #202

Earlier quoted context omitted.

Does this mean we’ve reached the next AI winter? This is as good as it gets for quite a long time? Honest question :) perhaps this will postpone everyone’s fears about the singularity…

Many years ago, there was an image that floated around with Craigslist and all the websites that replaced small parts of it—personals, for sale ads, etc. It turned out the way to beat Craigslist wasn’t to build Yet Another Monolithic Craigslist, but to chunk it off in pieces and be the best at that piece. This is analogous to what’s happening with AI models. Sam Altman is saying we have reached the point where spendi…

> Sam Altman is saying we have reached the point where spending $100M+ trying to “beat” GPT-4 at everything isn’t the future.

I don't disagree, but it does align pretty well with the OpenAI business model, no? "No need to develop your own base model, just buy our own"

Re: OpenAI’s CEO says the age of giant AI models is already over

#388

Post by dragonwriter »

DesiLurker wrote: post #386
fergie wrote: post #3

The most comforting AI news I have read this year.

I agree, probably for a different reason. IMO the fact that Sam is saying this likely means that the LLMs are close to the upper knee of the S-Curve and after GPT5 they likely dont have many new fundamental ideas (additional SCurves) to throw at it. since they are ahead of the rest, it likely means we might be headed for an AI improvement pause for couple of years after GPT5. Still good enough to upset the balance in…

> it likely means we might be headed for an AI improvement pause for couple of years after GPT5.

I suspect that a pause in base LLM performance won’t be an AI improvement pause; there’s a whole lot of space to improve the parts of AI systems around the core “brain in a jar” model.

Re: OpenAI’s CEO says the age of giant AI models is already over

#389

Post by DesiLurker »

jackmott42 wrote: post #9
fergie wrote: post #3

The most comforting AI news I have read this year.

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild

I actually have high hopes for the hybrid architecture Ben Goertzel has been working on at OpenCog. I think the LLMs are soon going to hit a S curve w/o introduction of additional scientific knowledge like physics and notion of energy (wrt AGI development, they'll still be good for tonnes of other jobs displacing things).

Re: OpenAI’s CEO says the age of giant AI models is already over

#390

Post by jabradoodle »

Art9681 wrote: post #261
Barrin92 wrote: post #248

Earlier quoted context omitted.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

GPT4: In the sentence "What is the third letter in the third word in this sentence?", the third word is "third." The third letter in "third" is 'i'. Given that we have only provided a text prompt and no context or body language hints, GPT seems to interpret this literally. The answer is correct. The third letter of the word "third" is "i".

That is not a literal interpretation, it is an incorrect one. Notice your need to use different language in your explanation.
Post reply on HN