Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

171–180 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#171

Post by spaceman_2020 »

treis wrote: post #119
spaceman_2020 wrote: post #94

Earlier quoted context omitted.

What about all the siloed content kept inside corporate servers? You won't get normal GPT to train on it, of course, but IBM could build a "IBM-bot" that has all the GPT-4 dataset + all of IBM's internal data. That model might be very well tuned to solve IBM's internal problems.

I don't think you can just feed it data. You've got to curate it, feed it to the LLM, and then manually check/further train the output. I also question that most companies have the volume and quality of data worth training on. It's littered with cancelled projects, old products, and otherwise obsolete data. That's going to make your LLM hallucinate/give wrong answers. Especially for regulated and otherwise legally en…

Well, IBM has 350k employees. If training a LLM on curated data costs tens of millions of dollars but ends up reducing headcount by 50k, it would be a massive win for any CEO.

You have to understand that all the incentives are perfectly aligned for corporations to put this to work, even spending tens of millions in getting it right.

The first corporate CEO who announces that his company used AI to reduce employee costs while increasing profits is going to get such a fat bonus that everyone will follow along.

Re: OpenAI’s CEO says the age of giant AI models is already over

#172

Post by bagels »

enduser wrote: post #47

"When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory." Bill Gates

Gates has refuted saying this. Are you implying by analogy that Altman hasn't said/will disclaim saying that "the age of giant AI models is almost over"?

Re: OpenAI’s CEO says the age of giant AI models is already over

#173

Post by dopidopHN »

mstolpm wrote: post #164
transcriptase wrote: post #115

Earlier quoted context omitted.

I had a fun conversation (more like argument) with ChatGPT about the hypocrisy of OpenAI. It would explicitly contradict itself and then began starting every reply with “I can see why someone might think…” and then just regurgitating fluff about democratizing AI. I finally was able to have it define democratization of technology and then recognize the absurdity of using that label to describe a pivot to gating models…

Why are you discussing OpenAI with ChatGPT? I’m honestly interested. I would imagine that any answer of ChatGPT on that topic is either (a) „hallucinated“ and not based on any verifiable fact or (b) scripted in by OpenAI. The same question pops up for me whenever someone asks ChatGPT about the internals and workings of ChatGPT. Am I missing something?

I’ve try because it’s tempting and the first attempts do give a « conversation » vibe.

I was curious about state persistence between prompt, or how to get my prompt better, or having a idea of the training data.

Only got crap and won’t spend time doing that again

Re: OpenAI’s CEO says the age of giant AI models is already over

#174

Post by eukara »

nabnob wrote: post #146
cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Real answer? Buy proprietary data from social media companies, credit card companies, retail companies and train the model on that data.

Can't wait for us to be able to query GPT for peoples credit card info

Re: OpenAI’s CEO says the age of giant AI models is already over

#175

Post by spaceman_2020 »

yunwal wrote: post #99
spaceman_2020 wrote: post #90

Earlier quoted context omitted.

Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…

But we're still nowhere near that, or even near surpassing the skill of an average person at a moderately complex information task, and GPT-4 supposedly took hundreds of millions to train. It also costs a decent amount more to run inference on it vs. 3.5. It probably makes sense to prove the concept that generative AI can be used for lots of real work before scaling that up by another order of magnitude for potential…

There are a lot of employees at large tech consultancies that don't really do anything that can't be automated away by even current models.

Sprinkle in some more specific training and I can totally see entire divisions at IBM and Accenture and TCS being made redundant.

The incentive structures are perversely aligned for this future - the CEO who manages to reduce headcount while increasing revenue is going to be very handsomely rewarded by Wall Street.

Re: OpenAI’s CEO says the age of giant AI models is already over

#176

Post by midland_trucker »

fpgaminer wrote: post #162
cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

> Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore. Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models. Or augment the dataset using RLHF, which provides an "infinite" dataset to train LLMs on. Limited by the capabil…

> Dataset size is not relevant to predicting the loss threshold of LLMs. You can keep pushing loss down by using the same sized dataset, but increasingly larger models.

Deepmind and others would disagree with you! No-one really knows in actual fact.

[1] https://www.deepmind.com/publications/an-empirical-analysis-...

Re: OpenAI’s CEO says the age of giant AI models is already over

#177

Post by moffkalast »

dragonwriter wrote: post #140
smeagull wrote: post #137

Earlier quoted context omitted.

This tells me you haven't really stress tested the model. GPT is currently at the stage of "person who is at the meeting, but not really paying attention so you have to call them out". Once GPT is pushed, it scrambles and falls over for most applications. The failure modes range from contradicting itself, making up things for applications that shouldn't allow it, to ignoring prompts, to simply being unable to perform…

Are we talking about bare GPT through the UI, or GPT with a framework giving it access to external systems and the ability to store and retrieve data? Because, yeah, “brain in a jar” GPT isn’t enough for most tasks beyond parlor-trick chat, but being used as a brain in a jar isn’t the point.

Still waiting to see those plugins rolled out and actual vector DB integration with GPT 4, then we'll see what it can really do. Seems like the more context you give it the better it does, but the current UI really makes it hard to provide that.

Plus the recursive self prompting to improve accuracy.

Re: OpenAI’s CEO says the age of giant AI models is already over

#178

Post by gumballindie »

Bruv has to pay for the data he's been using or soon there won't be any to nick on. Groupies claiming their ai is "intelligent", and not just a data ingesting beast, will soon learn a heard lesson. Take your blogs offline, stop contributing content for free and stop pushing code or else chavs like this one will continue monetising your hard work. As did bezos and many others that now want you to be out of a job.

Re: OpenAI’s CEO says the age of giant AI models is already over

#179

Post by xipix »

fauxpause_ wrote: post #141
summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Perhaps (a) there do exist bigger models that weren't better or (b) this model isn't better than somewhat smaller ones. Perhaps the CEO has seen diminishing returns.

Re: OpenAI’s CEO says the age of giant AI models is already over

#180

Post by moffkalast »

sailingparrot wrote: post #155
bagels wrote: post #144

Earlier quoted context omitted.

$5M to train GPT-4 is the best investment I've ever seen. I've seen startups waste more money for tremendously smaller impact.

As I stated in my comment, $5M is assuming you can do a much much better job than OpenAI at optimizing your training, only need to make a single training run, your employees salaries are $0, and you get a clean dataset for essentially free. Real cost is 10-20x that. That's still a good investment though. But the issue is you could very well sink $50M into this endeavour and end up with a model that actually is not re…

> and end up with a model that actually is not really good and gets rendered useless

ahem Bard ahem

Post reply on HN