Sounds like we will be on gpt4 for a while, while we wait out a new “AI winter”, mind you there is still a lot to squeeze from gpt4 and local/edge llm models
OpenAI’s CEO says the age of giant AI models is already over
331–340 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#332Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Perhaps (a) there do exist bigger models that weren't better or (b) this model isn't better than somewhat smaller ones. Perhaps the CEO has seen diminishing returns.
Sure. It is possible there is evidence that is not shared. It’s dumb to assume this is the case though.
Re: OpenAI’s CEO says the age of giant AI models is already over
#333Earlier quoted context omitted.
I think people said the same thing about NNs in general before we hit a scale where they started performing magic. There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant.
> There could be exponential or quadratic scaling laws with any of these black boxes that makes one approach suddenly extremely viable or even dominant. The reason I like the CPU approach is the memory scaling is bonkers compared to GPU. You can buy a server that has 12TB of DRAM (in stock right now) for the cost of 1 of those H100 GPU systems. This is enough memory to hold over 3 trillion parameters with full 32-bit…
Is your current prototype open source?
Re: OpenAI’s CEO says the age of giant AI models is already over
#334Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
It's not a wild claim when you have empirically well-validated scaling laws which make this very prediction.
That’s a generous usage of the term “empirically well-validated” and “law”
Re: OpenAI’s CEO says the age of giant AI models is already over
#335Earlier quoted context omitted.
Exponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query. Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.
Compute will also scale exponentially in coming years. The data source limitation seems to be a harder barrier, I think many companies are experimenting with AI generated content for training at this point.
> Compute will also scale exponentially in coming years.
Cost per transistor scaling has already plateaued or perhaps even inverted with TSMC's latest and greatest.
And the new chips, even after 25 layers of EUV lithography, more than doubling the previous record, and an extra year of fine tuning, has total SRAM size scaling of -5% and logic scaling of -42%.
These are numbers verified by experienced semi people.
Re: OpenAI’s CEO says the age of giant AI models is already over
#336Earlier quoted context omitted.
How are they at human performance? Almost everything GPT has read on the internet didn‘t even exist 200 years ago and was invented by humans. Heck, even most of the programming it does wasn‘t there 20 years ago. Not every programmer starting from scratch would be brilliant, but many were self taught with very limited resources in the 80s form example and discovered new things from there. GPT cannot do this and is ver…
>How are they at human performance? Because it performs at least average human level (mostly well above average) on basically every task it's given. "Invest something new" is a nonsensical benchmark for human level intelligence. The vast majority of people have never and will never invent anything new. If your general intelligence test can't be passed by a good chunk of humanity then it's not a general intelligence t…
> Because it performs at least average human level (mostly well above average) on basically every task it's given.
No, GPT4 fails at some very basic tasks. It can't count letters passed 15.
Re: OpenAI’s CEO says the age of giant AI models is already over
#337Earlier quoted context omitted.
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
"The third word of this sentence is "the," and its third letter is "e." - GPT-4
Unfortunately it seems clear that openai trains gptX on common test questions. They still fail novel ones.
Re: OpenAI’s CEO says the age of giant AI models is already over
#338"The great supercomputer Deep Thought is the most powerful computer ever built, with one exception.
It was designed by hyper-intelligent pan-dimensional beings, who wanted to know the answer to Life, the Universe, and Everything.
Its creation annoyed a fair few philosophers, who felt that it was taking over their turf.
After seven and a half million years of serious cogitation, Deep Thought spoke the answer. However, it was so inexplicable that Deep Thought then had to go on and design the most powerful computer ever built (with no exceptions) to work out what the question was."
Re: OpenAI’s CEO says the age of giant AI models is already over
#339The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
> (including Google itself!)
Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can.
Open collaboration in AI, at least when it comes to corporations, might have come to an end.
Re: OpenAI’s CEO says the age of giant AI models is already over
#340The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
[deleted]