91–100 of 525 posts
Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
This also makes me doubt that NSA hasn't already cracked this problem. Or that China won't eventually beat current western models since it will likely have way more data collected from its citizenry.
Humans are comprised of various "subnets" modelling aspects which, in unison, produce self-conciousness and real intelligence. What is missing in the current line of approaches is that we only rely on auto-alignment of subnetworks by machine learning, which scales only up to a point.
If we would produce a model which has
* something akin a LLM as we know it today, which is able to
* store or fetch facts to a short- ("context") or longterm ("memory") storage
* if not in the current "context", query the longterm context ("memory") by keywords for associations, which are one-by-one inserted into the current "context"
* repeat as required until fulfilling some self-defined condition ("thinking")
To me, this is mostly mechanical plumbing work and lots of money.
Also, if we get rid of the "word-boundedness" of LLMs - which we already did to some degree, as shown by the multi-language capabilities - LLMs would be free to roam in the domain of thoughts /s :)
This approach could be further improved by meta-LLMs governing the longterm memory access, providing an "intuition" which longterm memory suits the provided context best. Apply recursion as needed to improve results (paying by exponential training time, but this meta-NN will quite probably be independent of actual training, as real life / brain organization shows).
Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
He said that we are at the end of the era where capability improvements come primarily from making models bigger. Which stands to reason... I don't think anyone expect us to hit 100T parameters or anything.
Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
You can transcribe all spoken words everywhere and keep the model up to date? Keep indexing new data from chat messages, news articles, new academic work etc. The data is not finite.
That model might be very well tuned to solve IBM's internal problems.
Unless they can somehow keep their improvements ahead of the rest of the industry then they'll be lost among a crowd.
Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
Sam is now saying there will be no future model that will be as good.
This is all positioning to get regulators off the track because none of these control freaks in government actually understand a whit of this.
All said and done, this all just to try to disempower the OSS community. But they can't, we're blowing past their barriers like the 90s did with the definition of slippery slope.
Eh. Altman has a financial incentive to lie and obfuscate about what it takes to train a model like GPT-4 and beyond, so his word is basically worthless.
It could also be argued that there's financial incentive to just saying "giving us more money to train bigger models = better AI" forever
That's my theory and if I was a tech CEO in any of the companies competing in this space, that is what I would plan for.
Training an LLM will be the easy part going forward. It's building an ecosystem around it and hooking it up to everything that will matter. OpenAI will focus on this, while not-so-secretly training their next iterations.
>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…
Is cost really that much of a burden? Intelligence is the single most expensive resource on the planet. Hundreds of individuals have to be born, nurtured, and educated before you might get an exceptional 135+ IQ individual. Every intelligent person is produced at a great societal cost. If you can reduce the cost of replicating a 135 IQ, or heck, even a 115 IQ person to a few thousand dollars, you're beating biology b…
Also, just in terms of where to put your effort, if you think another direction (for example, fine-tuning the model to use digital tools, or researching how to predict confidence intervals) is going to have a better chance of success, why focus on scaling more?
>"the company’s CEO, Sam Altman, says further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” So to reiterate, he is not saying that the age of giant AI models is over. Current top-of-the-line AI models are giant and likely…
2 months since gpt 4.
This ride has only just started, fasten your whatevers.