Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s CEO says the age of giant AI models is already over

wired.com

Post reply on HN
Print view View original

141–150 of 525 posts

Re: OpenAI’s CEO says the age of giant AI models is already over

#141

Post by fauxpause_ »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.

Re: OpenAI’s CEO says the age of giant AI models is already over

#142

Post by [deleted] »

hanselot wrote: post #34
pixl97 wrote: post #6

The Age of Giants is over... The Age of Behemoths has begun! but sir, that means the same thing Throw this heretic into the pit of terror.

The pit of terror is full. Fine, to the outhouse of madness then. Before I get nuked from orbit for daring to entertain humor, if someone is running ahead of me in a marathon, and running so far ahead, yet still broadcasting things to the back for the slow people (like myself), then eventually we catch up to them, and they suddenly say, you know what guys, we should stop running in this direction, there's nothing to…

[deleted]

Re: OpenAI’s CEO says the age of giant AI models is already over

#143

Post by bitL »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

Transformers were known that they kept scaling up with more parameters and more training data so if the Open AI hit the limits of this scaling that would be a very important milestone in AI.

Re: OpenAI’s CEO says the age of giant AI models is already over

#144

Post by bagels »

sailingparrot wrote: post #125
ldehaan wrote: post #100

Earlier quoted context omitted.

I've been training large 65b models on "rent for N hours" systems for less than 1k per customized model. Then fine tuning those to be whatever I want for even cheaper. 2 months since gpt 4. This ride has only just started, fasten your whatevers.

Finetuning cost are nowhere near representative of the cost to pre-train those models. Trying to replicate the quality of GPT-3 from scratch, using all the tricks and training optimizations in the books that are available now but weren't used during GPT-3 actual training, will still cost you north of $500K, and that's being extremly optimistic. GPT-4 level model would be at least 10x this using the same optimism (mea…

$5M to train GPT-4 is the best investment I've ever seen. I've seen startups waste more money for tremendously smaller impact.

Re: OpenAI’s CEO says the age of giant AI models is already over

#145

Post by Spivak »

Art9681 wrote: post #98
hbn wrote: post #85

Earlier quoted context omitted.

It could also be argued that there's financial incentive to just saying "giving us more money to train bigger models = better AI" forever

I don't think these comments are driven from financial incentives. It's a distraction and only a fool would believe Altman here. What this likely means is they are prioritizing adding more features to their current models while they train the next version. Their competitors scramble to build an LLM with some sort of intelligence parity, when that happens no one will care because ChatGPT has the ecosystem and plugins…

text-davinci-003 but cheaper and runs on your own hardware is already a massive selling point. If you you release a foundational model at parity with GPT4 you'll win overnight because OpenAI's chat completions are awful even with the super advanced model.

Re: OpenAI’s CEO says the age of giant AI models is already over

#146

Post by nabnob »

cleandreams wrote: post #14

Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.

Real answer? Buy proprietary data from social media companies, credit card companies, retail companies and train the model on that data.

Re: OpenAI’s CEO says the age of giant AI models is already over

#147

Post by sebzim4500 »

it wrote: post #132

Interesting how this contradicts "The Bitter Lesson": http://incompleteideas.net/IncIdeas/BitterLesson.html .

I don't think there is a contradiction at all. Altman is essentially saying they are running out of compute and therefore can't meaningfully scale further. Not that scaling further would be a worse plan longterm than coming up with new algorithms.

Re: OpenAI’s CEO says the age of giant AI models is already over

#148

Post by cloudking »

sgu999 wrote: post #121

Is anyone aware of techniques to prune a model from useless knowledge to leave more space for the reasoning capabilities? It really shouldn't matter that it can give the exact birthdate of Steve Wozniac, as long as it can properly make a query to fetch it and deal with the result.

I follow your design, couldn't you also solve hallucinations with a "fact checking" LLM (connected to search) that corrects the output of the core LLM? You would take the output of the core LLM, send it to the fact checker with a prompt like "evaluate this output for any potential false statements, and perform an internet search to validate and correct them"

Re: OpenAI’s CEO says the age of giant AI models is already over

#149

Post by GaggiX »

summerlight wrote: post #135

The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…

I think the next step is multimodality, GPT-4 can "see" probably using a method similar to miniGPT-4, so the embeddings are aligned using Q-former (or something similar), the next step would be to actually predict image tokens using the LM loss, this way it would be able to use the knowledge gained by "seeing" on other tasks like: making actual good ASCII art, making SVG that makes sense, and on a less superficial level having a better world model.

Re: OpenAI’s CEO says the age of giant AI models is already over

#150

Post by Gatsky »

boringuser2 wrote: post #78

Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.

Do you think GPT-4 was trained and then immediately released to the public? Training finished Aug 2022. They spent the next 6 months improving it in other ways (eg human feedback). What he is saying is already evident therefore.
Post reply on HN