Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

241–250 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#241

Post by narrator »

cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

You could have it start talking to itself in the way that AlphaGO learns to get better at Go. All that needs to be done is find some fitness function that indicates that useful knowledge has been produced. In Go and Chess this is easy.

It can start posting synthesized ideas on social media and see how many likes it gets. Coupled with a metric containing dissimilarity to current information, this could be a useful way to progress to superhuman insights.

Re: OpenAI’s CEO says the age of giant AI models is already over

#242

Post by sdenton4 »

bob1029 wrote: post #151

I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…

Meh, people have been trying to ape biological networks for decades and they consistently produce poor functioning versions of the leading edge... Why wouldn't we have already seen spiking networks contribute something by now?

Re: OpenAI’s CEO says the age of giant AI models is already over

#243

Post by hesdeadjim »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a gigantic increase in the context window would be a great next step. GPT-4 is so damned good I tend to only become disappointed when it is forced to forget history.

Re: OpenAI’s CEO says the age of giant AI models is already over

#245

Post by lannisterstark »

transcriptase wrote: post #115
cowpig wrote: post #82

Earlier quoted context omitted.

OpenAI has gone from open-sourcing its work, to publishing papers only, to publishing papers that omit important information, to GPT-4 being straight-up closed. And Sam Altman doesn't exactly have a track record of being overly concerned about the truth of his statements.

I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models…

"Then it basically told me “well it’s for safety and protecting society”."

That was pretty much OpenAI's argument when they first published that GPT-3 paper. "Oh no so scary people might use it for wrong stuff, only we should have control of it."

In reality they just saw $$$ blink.

Re: OpenAI’s CEO says the age of giant AI models is already over

#246

Post by cjbprime »

nomel wrote: post #194
mnky9800n wrote: post #189

Earlier quoted context omitted.

or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…

> model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data…

GPT-4 is actually multi-modal, not text. ChatGPT does not yet expose image upsubmission to it. But it's part of how the model was trained already.

Re: OpenAI’s CEO says the age of giant AI models is already over

#247

Post by beachy »

ericmcer wrote: post #181
sitkack wrote: post #88

Earlier quoted context omitted.

5 cents. Per resume. $500 per 10k. 1-3 hours of a fully loaded engineers salary per year. You are being criminally cheap.

For a big company that is nothing but if you are bootstrapping and trying to acquire customers with an MVP racking up a $500 bill is frightening. What if you offer a free trial and blow up and end up with 5k+ bill.

By these maths, the $500 bill is for 10K resumes.

To show an MVP to a customer you only need 10 resumes (or 1 in most demos I've been in).

So 50c.

Re: OpenAI’s CEO says the age of giant AI models is already over

#248

Post by Barrin92 »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Both ChatGPT 3.5 and 4 literally fail the question:

"What is the third letter in the third word of this sentence"

When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path

Re: OpenAI’s CEO says the age of giant AI models is already over

#249

Post by nwlieb »

AlecSchueler wrote: post #239
mensetmanusman wrote: post #235

Earlier quoted context omitted.

Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.

Don't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.

The runtime is quadratic for a given context size, although it seems like there is some progress on this front https://gwern.net/note/attention

Re: OpenAI’s CEO says the age of giant AI models is already over

#250

Post by nuc1e0n »

ldehaan wrote: post #97

This is just push back from elon and crews fake article about the dangers of AI, they specifically state the next versions will be ultra deadly. Sam is now saying there will be no future model that will be as good. This is all positioning to get regulators off the track because none of these control freaks in government actually understand a whit of this. All said and done, this all just to try to disempower the OSS…

He isn't saying that. Only that different approaches are needed to produce better results. If he's confident to say this I think he must have something up his sleeve.
Post reply on HN