"When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory." Bill Gates
That was in context of a how much RAM would be needed for software of that era, not the entire future of computing.
OpenAI’s CEO says the age of giant AI models is already over
431–440 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#432"When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory." Bill Gates
Gates has refuted saying this. Are you implying by analogy that Altman hasn't said/will disclaim saying that "the age of giant AI models is almost over"?
Just that there is tremendous hubris in the statement—at least when the statement stands alone. Vastly larger LLMs will probably become one or more relatively small components or layers of much larger systems that run on whatever we use as telephones in 20 years.
I suspect what he means is that OpenAI is finding diminishing returns from throwing money and hardware at larger models right now and that they are investigating other and/or composite AI techniques that make more optimal use of their hardware investment.
Re: OpenAI’s CEO says the age of giant AI models is already over
#433The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
> It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future.
Given the amount of resources being thrown at AI right now, i consider this to be very unlikely indeed.
Re: OpenAI’s CEO says the age of giant AI models is already over
#434The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
> It's likely that GPT-4 is on the optimal spot between cost and performance and there won't be significant improvements on performance in a near future. Given the amount of resources being thrown at AI right now, i consider this to be very unlikely indeed.
I think GP's comment is probably apt if understood together with the qualification of "unless we find a better modeling architecture than Transformer".
Re: OpenAI’s CEO says the age of giant AI models is already over
#435>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…
I've been training large 65b models on "rent for N hours" systems for less than 1k per customized model. Then fine tuning those to be whatever I want for even cheaper. 2 months since gpt 4. This ride has only just started, fasten your whatevers.
You are confusing training with fine-tuning which is a different beast.
Re: OpenAI’s CEO says the age of giant AI models is already over
#436I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run. Pretty sure Microsoft swapped out Bing for something a lot smaller in the last couple of weeks; Google hasn't even tried to implement a publicly available large model. And OpenAI still has usage caps on their GPT-4. I'd bet that they can still see improvement in performance with GPT-5, but that when they look a…
> I'd bet that what he, and the competition, is realizing is that the bigger models are too expensive to run. I think it's likely that they're out of training data to collect. So adding more parameters is no longer effective. > most people don't need a brilliantly intelligent assistant, they just need a not-dumb assistant. I tend to agree, and I think their pathway toward this will all come from continuing advances i…
Once they can add videos they will have a lot of new training data.
Re: OpenAI’s CEO says the age of giant AI models is already over
#437Earlier quoted context omitted.
>How are they at human performance? Because it performs at least average human level (mostly well above average) on basically every task it's given. "Invest something new" is a nonsensical benchmark for human level intelligence. The vast majority of people have never and will never invent anything new. If your general intelligence test can't be passed by a good chunk of humanity then it's not a general intelligence t…
Yeah these intelligence tests are not very good. I would argue some programmers do in fact invent something new. Not all of them, but some. Perhaps 10%. Second the point is not whether everyone is by profession an inventor but whether most people can be inventors. And to a degree they can be. I think you underestimate that by a large margin. You can lock people in a room and give them a problem to solve and they will…
> Not all of them, but some. Perhaps 10%.
Just listen to what you're saying:
- GPT isn't at human level because GPT isn't able to invent something new
- Not all programmers invent something new, but some. Perhaps 10%
I'm pretty sure this implies literally that 90% programmers aren't human level.
The lengths to which people are willing to go to dismiss GPT's abilities is mind boggling to me.
Re: OpenAI’s CEO says the age of giant AI models is already over
#438Earlier quoted context omitted.
In this case I think it's Wired that's lying. Altman didn't say large models have no value, or that there will be no more large models, or that people shouldn't invest in large models. He said that we are at the end of the era where capability improvements come primarily from making models bigger. Which stands to reason... I don't think anyone expect us to hit 100T parameters or anything.
What about 1T though, seems silly to stop here.
Like Altman said, it's comparable to the GHz race in the 1990's. If 4GHz is good, 5GHz is better, why not 10GHz?
Turns out there are diminishing returns and advances come from other dimensions. I've got no opinion on whether he's right or not, but he's certainly in a better position to opine that current scale has hit diminishing returns.
In any event, there's nothing special about 1T parameters. It's just a round base-10 number. It is no more magic than 900B or 1.3T.
Re: OpenAI’s CEO says the age of giant AI models is already over
#439Earlier quoted context omitted.
NewPipe/Freetube/Invidious instances + SponsorBlock API support is really great at filtering all of these useless advertising memes. Somehow ($$$) the behavior of TV adverts culture seeped into YouTube. I've not watched broadcast/cable TV in years, but when I interact with people that do, inevitably, they make similar references. It's super weird.
Wait... You can auto skip the in-video sponsor ads?
Yeah, SponsorBlock (https://sponsor.ajay.app/) is crowdsourced data for the in-video ads. Weirdly enough, I've never contributed timestamps, but the vast majority of the content I watch has submitted timestamps. I support a bunch of creators on patreon, but very much disagree with their double dipping.