It's not clear from the article whether it's a dense model or MoE. This matters when it comes to comparing with GPT-4 - in terms of # params - which is reported to be MoE
Training for one trillion parameter model backed by Intel and US govt has begun
21–30 of 267 posts
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#22Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…
Your profile says "I am not rabidly against technology, but I promote a BALANCED, severely critical look at it" "Humanity should give up this useless development of AI" So balanced.
I do not mean balanced against every technology. Some technologies might have some merit.
I do believe that AI has no merit, that people developing it are doing is a disservice, and that it should be completely destroyed. OF COURSE, there are some technologies that I think are completely bad, including chemical weapons. In fact, I believe AI to be on the level of chemical weapons essentially.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#23The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
> "No human requires the amount of training data we are already giving the models."
Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on.
Maybe the ideal way to train an AI is to have it interact with lots of humans, so it can try things on its own? The implication of that is maybe the best trained AI will be the center of some important web property, like, say, the Google search engine (I'm imagining something like Google search now, but more conversational -- it asks you if that was what you're looking for, and asks clarifying questions.) Whoever has the most users will have the best AI, which creates the best product, which attracts the most users... and so on.
I do agree that architectural improvements could be hugely significant too.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#24The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
> "No human requires the amount of training data we are already giving the models." Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on. Maybe the ideal way to train an AI is to have it interact with lots of humans, so…
The first humans were kinda dumb. Yet by interacting with the world and each other, humans got smart. I wonder if neural network instances can interact with a simulated environment and each other, they could become smart enough for practical purposes.
Kinda like alpha go zero but with "life".
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#25Not surprising. Despite the enormous energy costs and the threat to humanity by creating technology that we can't control, governments and corporations will build bigger and more sophisticated models just because they have to do so to compete. It is the prisoner's dilemma: we end up with a super-advanced AI that will disrupt and make society worse because entities are competing where the metric of success is short-te…
Your profile says "I am not rabidly against technology, but I promote a BALANCED, severely critical look at it" "Humanity should give up this useless development of AI" So balanced.
Having a balanced view about technology doesn't mean considering every type of technology to be 50% good and 50% bad.
I disagree with their view on AI but also disagree with your criticism.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#26The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
[deleted]
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#27Earlier quoted context omitted.
Your profile says "I am not rabidly against technology, but I promote a BALANCED, severely critical look at it" "Humanity should give up this useless development of AI" So balanced.
I do not mean balanced against every technology. Some technologies might have some merit. I do believe that AI has no merit, that people developing it are doing is a disservice, and that it should be completely destroyed. OF COURSE, there are some technologies that I think are completely bad, including chemical weapons. In fact, I believe AI to be on the level of chemical weapons essentially.
Can you elaborate on this view? Chemical weapons seem to have no redeeming qualities. LLMs have plenty -- I use OpenAI-backed products at work quite regularly, and find they've increased my productivity and my general happiness with my job.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#28Earlier quoted context omitted.
> "No human requires the amount of training data we are already giving the models." Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on. Maybe the ideal way to train an AI is to have it interact with lots of humans, so…
The first humans were kinda dumb. Yet by interacting with the world and each other, humans got smart. I wonder if neural network instances can interact with a simulated environment and each other, they could become smart enough for practical purposes. Kinda like alpha go zero but with "life".
> The first humans were kinda dumb
How can you assert this ? Do you have any evidence?
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#29The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
[deleted]
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#30Earlier quoted context omitted.
The first humans were kinda dumb. Yet by interacting with the world and each other, humans got smart. I wonder if neural network instances can interact with a simulated environment and each other, they could become smart enough for practical purposes. Kinda like alpha go zero but with "life".
> The first humans were kinda dumb How can you assert this ? Do you have any evidence?
Can't tell if you're joking or serious.
In the latter case, the first humans (H. Habilis) had about 1/2 of H. Sapiens brain to work with, and a much smaller fraction of neocortex.
If that doesn't satisfy you, let's say I was speaking about some sort of human ancestor before that, which would have been about as dumb as chimps, unless you require proof of their dumbness as well.