Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

411–420 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#411

Post by kvetching »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

I think there's a single point that we need to keep an eye out for;

A model that got to the point where it's possible for it to propose new architectures, improve optimization & efficiency.

In other words, the extremely massive model that could do this doesn't need to be available to the public. It's sole purpose should be to be used internally by a company to improve itself.

This is actually the point at which many say could lead to the singularity.

Re: OpenAI’s CEO says the age of giant AI models is already over

#412

Post by Ambix »

nomel wrote: post #194
mnky9800n wrote: post #189

Earlier quoted context omitted.

or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…

> model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data…

> "a picture is worth a thousand words"

and it might be opposite for the GPT models actually. it's just easier for humans to grasp the bunch of knowledge with one eyes sight, but usually most of useful information might be represented with just of bunch of words and machines are to scan through the millions of words in an instant.

Re: OpenAI’s CEO says the age of giant AI models is already over

#413

Post by PartiallyTyped »

ActorNightly wrote: post #399
PartiallyTyped wrote: post #382

Earlier quoted context omitted.

There's only so much first order derivatives can do for you. My bet is on sparsity, lottery tickets and symmetries.

If you think about it, Transformers were basically a way to just generalize convolution - instead of a fixed kernel shape in the sense of image processing, you now have a learned kernel arbitrary shape. Big advancement in terms of what they allowed, but fundamentally not really a new concept. While these things represent a fundamental way we store information as humans, these have very little to do with actual reason…

If we wanted to generalize this further, attention is 'just' an instance of graph convolution, encoder-only models like Bert are complete graphs, decoder-only models like GPT are still complete graphs but the information flow is not bi-directional as in Bert; instead a node provides information to all subsequent nodes only giving rise to the causal nature.

However, I don't think Hebbian learning will see a resurgence except maybe if it motivates some kind of pruning mechanism.

I think that Sutton was right in 'The bitter lesson', the problem seems to be that we are hitting the limits of what we can do with our compute.

Re: OpenAI’s CEO says the age of giant AI models is already over

#414

Post by Aerbil313 »

taylorius wrote: post #169
dauertewigkeit wrote: post #138

Earlier quoted context omitted.

Are intelligent people that valuable? There's lots of them at every university working for peanuts. They don't seem to be that valued by society, honestly.

IQ isn't all that. Mine is 140+ and I'm just a somewhat well paid software engineer. It's TOO abstract a metric in my view - for sure it doesn't always translate into real world success.

Mine is 150-160 (varies by how much I’m sleep deprived during the IQ test) and I’m told that I’m exceptionally intelligent by teachers, friends, colleagues, most everyone I met since early childhood. I guess the more the difference to average, the more it stands out. From my experience, I believe higher IQ is nothing but better pattern recognition and being smart or genius means merely higher IQ + very good memory capability. I believe those two are interlinked [0]. By memory capability I mean not forgetting anything you’ve ever seen, not in the sense of being able to recall every minute of your entire life, but in the sense of reliably and always being able to recall all info regarding X if you’ve seen X only once in your life. Higher intelligence doesn’t mean you’re automatically better off than everyone also. It just means that you can be far ahead of everyone in any cognitive task with a far smaller amount of effort put in. Note that this doesn’t conflict with IQ being a reliable predictor of financial success, rather I believe this is the reason for it.

[0]: https://saveall.ai/blog/learning-is-remembering

Re: OpenAI’s CEO says the age of giant AI models is already over

#415

Post by okwhateverdude »

hn_go_brrrrr wrote: post #394
heavyset_go wrote: post #351

Earlier quoted context omitted.

It took a year's worth of a small nation's energy expenditure to train and all it does is tell you how Raycon earbuds will change your life.

I have never been so pleased to not get any of these references.

NewPipe/Freetube/Invidious instances + SponsorBlock API support is really great at filtering all of these useless advertising memes. Somehow ($$$) the behavior of TV adverts culture seeped into YouTube. I've not watched broadcast/cable TV in years, but when I interact with people that do, inevitably, they make similar references. It's super weird.

Re: OpenAI’s CEO says the age of giant AI models is already over

#416

Post by ckdarby »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

If you use GPT4 as a "routing" service to send down to other models the journey is only beginning and most of the industry hasn't understood we're well beyond a doubling at the moment.

Re: OpenAI’s CEO says the age of giant AI models is already over

#417

Post by chaoz_ »

potamic wrote: post #404
worrycue wrote: post #339

Earlier quoted context omitted.

> (including Google itself!) Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can. Open collaboration in AI, at least when it comes to corporations, might have…

Doesn't Google have any patents on the transformer architecture? I assume large enterprises tend to patent everything that comes out of their research.

https://patents.google.com/patent/US10452978B2/en

GPT models are based on transformer, but architecture is different from what's patented.

Not a lawyer, but can you really patent certain network architecture? Theoretically someone could invent new activation function that just happens to make same architecture perform a lot better on some tasks, can you call really cover that with patent?

Re: OpenAI’s CEO says the age of giant AI models is already over

#418

Post by midland_trucker »

fpgaminer wrote: post #260
midland_trucker wrote: post #176

Earlier quoted context omitted.

> Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models. Deepmind and others would disagree with you! No-one really knows in actual fact. [1] https://www.deepmind.com/publications/an-empirical-analysis-...

I don't recall the Chinchilla paper disputing my point. They establish "training-compute optimal" scaling laws, but none of their findings suggest that loss hits any kind of asymptote.

Perhaps we're talking past each other, is "loss threshold" a specific term in LLM literature?

Merely pointing out that the debate as to whether we are compute or data limited (OP) has not concluded at all; There are lots of compelling theories on relationship between the two.

Re: OpenAI’s CEO says the age of giant AI models is already over

#419

Post by narrator »

forrest2 wrote: post #263
narrator wrote: post #233

"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them." An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are…

You might not see it available via self serve interfaces, but I think the h100 cards can do 256x to a single host.

You're right! You can nvlink 32 systems with 8 h100s! That's 20 terabytes of VRAM total!

Re: OpenAI’s CEO says the age of giant AI models is already over

#420

Post by Alifatisk »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

> while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop and took all the rewards.

You hit the nail

Post reply on HN