Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

261–270 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#261

Post by Art9681 »

Barrin92 wrote: post #248
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

GPT4: In the sentence "What is the third letter in the third word in this sentence?", the third word is "third." The third letter in "third" is 'i'.

Given that we have only provided a text prompt and no context or body language hints, GPT seems to interpret this literally. The answer is correct. The third letter of the word "third" is "i".

Re: OpenAI’s CEO says the age of giant AI models is already over

#262

Post by dumbmrblah »

htk wrote: post #258

Maybe they're now training teams of AIs to work together? This way the hardware limitations are less of a barrier in terms of maximum memory per AI. They can run in parallel in separate hardware and communicate to reach higher goals.

Someone described me a emerging rule called “Penelope‘s rule” where you scale by having the largest and most powerful model delegate to the less powerful models and then have the larger model improve upon the results. (Eg have GPT-4 delegate the more mundane tasks to GPT-3.5).

Re: OpenAI’s CEO says the age of giant AI models is already over

#263

Post by forrest2 »

narrator wrote: post #233

"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them." An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are…

You might not see it available via self serve interfaces, but I think the h100 cards can do 256x to a single host.

Re: OpenAI’s CEO says the age of giant AI models is already over

#264

Post by Barrin92 »

Art9681 wrote: post #261
Barrin92 wrote: post #248

Earlier quoted context omitted.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

GPT4: In the sentence "What is the third letter in the third word in this sentence?", the third word is "third." The third letter in "third" is 'i'. Given that we have only provided a text prompt and no context or body language hints, GPT seems to interpret this literally. The answer is correct. The third letter of the word "third" is "i".

>GPT seems to interpret this literally.

No it doesn't interpret anything. (which is why I've gotten three completely different answers here on HN within 10 minutes). It cannot count at all or even understand the question which becomes obvious when you ask it for word counts or letter counts that don't match anything in the sentence, at which point it makes up nonsense.

what is the tenth word in the following sentence: "Life is great!"

>"The tenth word in the sentence "Life is great!" is "great."

Re: OpenAI’s CEO says the age of giant AI models is already over

#265

Post by nirav72 »

Barrin92 wrote: post #248
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

I just tried this out with 4 and 3.5 - in both tests, it came back with the ‘third’ as the 3rd word and ‘i’ as the 3rd letter. It completely ignored ‘the’ as the 3rd word.

Re: OpenAI’s CEO says the age of giant AI models is already over

#266

Post by guardiangod »

Barrin92 wrote: post #248
fauxpause_ wrote: post #141

Earlier quoted context omitted.

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

This has been solved in the last year or so. Look up "LLM chain of thought".

Re: OpenAI’s CEO says the age of giant AI models is already over

#269

Post by siliconc0w »

It's probably economically difficult to rationalize building a GPT5. They're already having a hard time getting everyone GPT4. Better to spend the resources optimizing to get everyone onto a cheap(er) GPT4-comparable model than build a marginally better model that is even more expensive to scale. Like most tech there is a sigmoid curve where you start to see diminishing returns.

Re: OpenAI’s CEO says the age of giant AI models is already over

#270

Post by bhouston »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

I suspect you are right. We may be stuck at the gpt4 sizes for a bit just because of hardware costs though. As they get bigger it costs too much to run them until our hardware becomes more optimal for these large models at 4 bits or so.

I think the YouTube videos is going to be the next big training set. A transformer trained on all text and all of YouTube will be killer amazing at so much. I bet it can understand locomotion and balance and body control from YouTube.

I wonder if TPUs, like Google's Tensor chip, will beat out GPUs when it comes to image/video based training?

Post reply on HN