Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

51–60 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#51

Post by mullingitover »

thunderbird120 wrote: post #27

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…

Quality over quantity. Just building a model with a gazillion parameters isn't indicative of quality, you could easily have garbage parameters with tons of overfitting. It's like megapixel counts in cameras: you might have 2000 gigapixels in your sensor, but that doesn't mean you're going to get great photos out of it if there are other shortcomings in the system.

Re: OpenAI’s CEO says the age of giant AI models is already over

#52

Post by cj »

causi wrote: post #25
jackmott42 wrote: post #9

Earlier quoted context omitted.

If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild

I worry that the hardware requirements are only going to accelerate the cloud-OS integration. Imagine a PC that's entirely unusable offline.

> Imagine a PC that's entirely unusable offline.

FWIW we had thin clients in computer labs in middle school / high school 15 years ago (and still today these are common in enterprise environments, e.g. Citrix).

Biggest issue is network latency which is limited by the speed of light, so I imagine if computers in 10 years require resources not available locally it would likely be a local/cloud hybrid model.

Re: OpenAI’s CEO says the age of giant AI models is already over

#53

Post by brucethemoose2 »

tfehring wrote: post #20

Related reading: https://dynomight.net/scaling/ In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third. Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise…

Better data is still critical, even if bigger data isn't. The linked article emphasizes this.

Re: OpenAI’s CEO says the age of giant AI models is already over

#54

Post by [deleted] »

hcks wrote: post #46
thunderbird120 wrote: post #27

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…

Yes, but it also tells us that if Altman is honest here, then he doesn’t believe GPT-like models can scale to near level human performances (because even if the cost of compute was 10x or even 100x it would still be economically sound).

[deleted]

Re: OpenAI’s CEO says the age of giant AI models is already over

#55

Post by famouswaffles »

hcks wrote: post #46
thunderbird120 wrote: post #27

>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…

Yes, but it also tells us that if Altman is honest here, then he doesn’t believe GPT-like models can scale to near level human performances (because even if the cost of compute was 10x or even 100x it would still be economically sound).

No it doesn't.

For one thing they're already at human performance.

For another, i don't think you realize how expensive inference can get. Microsoft with no scant amount of available compute is struggling to run gpt-4 such that they're rationing it between subsidiaries while they try to jack up compute.

So saying, it would be economically sound if it cost x10 or x100 what it costs now is a joke.

Re: OpenAI’s CEO says the age of giant AI models is already over

#57

Post by [deleted] »

it_citizen wrote: post #50
machdiamonds wrote: post #41

Earlier quoted context omitted.

Ilya Sutskever (OpenAI Chief Scientist): "Yeah, I would say the data situation is still quite good. There's still lots to go" - https://youtu.be/Yf1o0TQzry8?t=685 There was a rumor that they were going to use Whisper to transcribe YouTube videos and use that for training. Since it's multimodal, incorporating video frames alongside the transcriptions could significantly enhance its performance.

I am curious how much video-to-text content represent compared to pure text. I have no idea.

[deleted]

Re: OpenAI’s CEO says the age of giant AI models is already over

#59

Post by haxton »

thewataccount wrote: post #48
xt00 wrote: post #7

Saying “hey don’t go down the path we are on, where we are making money and considered the best in the world.. it’s a dead end” rings pretty hollow.. like “don’t take our lunch please?” Might be a similar statement it feels..

Nah - GPT-4 is crazy expensive, paying 20$/mo only get's you 25messages/3hours and it's crazy slow. The api is rather expensive too. I'm pretty sure that GPT-4 is ~1T-2T parameters, and they're struggling to run it(at reasonable performance and profit). So far their strategy has been to 10x the parameter count every GPT generation, and the problem is that there's diminishing returns everytime they do that. AFAIK they…

I don't think this argument really holds up.

GPT3 on release was more expensive ($0.06/1000 tokens vs $0.03 input and $0.06 output for GPT4).

Reasonable to assume that in 1-2 years it will also come down in cost.

Re: OpenAI’s CEO says the age of giant AI models is already over

#60

Post by idiotsecant »

MuffinFlavored wrote: post #13
jcims wrote: post #12

I'm no expert but doesn't the architecture of minigpt4 that's on the front page right now give some indication of what the future might look like?

eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)

No? I use it all the time to help me, for example, read ML threads when I run into a term I don't immediately understand. I can do things like 'explain this at the level of a high school student'
Post reply on HN