Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

41–50 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#41

Post by machdiamonds »

cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685

There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.

Re: OpenAI’s CEO says the age of giant AI models is already over

#42

Post by Teever »

xwdv wrote: post #40
jackmott42 wrote: post #9

Earlier quoted context omitted.

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild

People think something magical happens when AI are wired together and give each other feedback. Really you’re still just predicting the next word, but with extra steps.

People think that something magical happens when transistors are wired together and give each other feedback.

Really you're just switching switches on and off, but with extra steps.

Re: OpenAI’s CEO says the age of giant AI models is already over

#43

Post by JoshuaDavid »

MuffinFlavored wrote: post #13
jcims wrote: post #12

I'm no expert but doesn't the architecture of minigpt4 that's on the front page right now give some indication of what the future might look like?

eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)

They're good for tasks where generation is hard but verification is easy. Things like "here I gesture at a vague concept that I don't know the name of, please tell me what the industry-standard term for this thing is" where figuring out the term is hard but looking up a term to see what it means is easy. "Create an accurate summary of this article" is another example - reading the article and the summary and verifying that they match may be easier than writing the summary yourself.

Re: OpenAI’s CEO says the age of giant AI models is already over

#44

Post by galaxytachyon »

idiotsecant wrote: post #16
galaxytachyon wrote: post #11

What age? Like, 3 years? On the other hand though, Chinchilla and multimodal approaches already showed how later AIs can be improved beyond throwing petabytes of data at them. It is all about variety and quality from now on I think. You can teach a person all about the color zyra but without actually ever seeing it, they will never fully understand that color.

It does seem, though, that using chinchilla like techniques does not create a copy with the same quality as the original. It's pretty good for some definition of the phrase, but it isn't equivalent, it's a lossy technique.

I agree on the lossy. There is a tradeoff between efficiency and comprehensiveness, kind of. It would be pretty funny if in the end, the most optimal method turns out to be the brain we already have. Extremely efficient, hardware optimized, but slow as hell and misunderstand stuff all the time unless prompted with specific phrases.

Re: OpenAI’s CEO says the age of giant AI models is already over

#45

Post by MattPalmer1086 »

MuffinFlavored wrote: post #13
jcims wrote: post #12

I'm no expert but doesn't the architecture of minigpt4 that's on the front page right now give some indication of what the future might look like?

eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)

Thing is, you can't trust what you find on stack overflow or other sources either. And searching, reading documentation and so on takes a lot of time too.

I've personally been using it to explore using different libraries to produce charts. I managed to try out about 5 different libraries in a day with fairly advanced options for each using chatGPT.

I might have spent a day in the past just trying one and not to the same level of functionality.

So while it still took me a day, my final code was much better fitted to my problem with increased functionality. Not a time saver then for me but a quality enhancer and I learned a lot more too.

Re: OpenAI’s CEO says the age of giant AI models is already over

#46

Post by hcks »

thunderbird120 wrote: post #27

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…

Yes, but it also tells us that if Altman is honest here, then he doesn’t believe GPT-like models can scale to near level human performances (because even if the cost of compute was 10x or even 100x it would still be economically sound).

Re: OpenAI’s CEO says the age of giant AI models is already over

#48

Post by thewataccount »

xt00 wrote: post #7

Saying “hey don’t go down the path we are on, where we are making money and considered the best in the world.. it’s a dead end” rings pretty hollow.. like “don’t take our lunch please?” Might be a similar statement it feels..

Nah - GPT-4 is crazy expensive, paying 20$/mo only get's you 25messages/3hours and it's crazy slow. The api is rather expensive too.

I'm pretty sure that GPT-4 is ~1T-2T parameters, and they're struggling to run it(at reasonable performance and profit). So far their strategy has been to 10x the parameter count every GPT generation, and the problem is that there's diminishing returns everytime they do that. AFAIK they've now resorted to chunking GPT through the GPUs because of the 2 to 4 terabytes of VRAM required (at 16bit).

So now they've reached the edge of what they can reasonably run, and even if they do 10x it the expected gains are less. On top of this, models like LLaMa have shown that it's possible to cut the parameter count substantially and still get decent results (albiet the opensource stuff still hasn't caught up).

On top of all of this, keep in mind that at 8bit resolution 175B parameters (GBPT3.5) requires over 175GB of VRAM. This is crazy expensive and would never fit on consumer devices. Even if you use quantization and use 4bit, you still need over 80GB of VRAM.

This definitely is not a "throw them off the trail" tactic - in order for this to actually scale the way everyone envisions both in performance and running on consumer devices - research HAS to be on improving the parameter count. And again there's lots of research showing its very possible to do.

tl;dr: smaller = cheaper+faster+more accessible+same performance

Re: OpenAI’s CEO says the age of giant AI models is already over

#50

Post by it_citizen »

machdiamonds wrote: post #41
cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685 There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.

I am curious how much video-to-text content represent compared to pure text. I have no idea.
Post reply on HN