I'm sure the government's mission is also to develop an AGI that benefits us all.
[deleted]
Training for one trillion parameter model backed by Intel and US govt has begun
171–180 of 267 posts
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#172Earlier quoted context omitted.
The opposite for me. Corporation have naked greed as their driving motivation, but that usually doesn't involve killing off all of their customers. That would be quite unprofitable. People elected to government often seem to seek power for powers sake and I have less faith that they'll not harm us.
I sure see a lot of power for power’s sake inside of companies. OpenAI’s stated mission is AGI that can replace half the population in “economically valuable” work. I get that with some creativity you can see that as a net benefit for humanity, but across at least a generation that’s going to be a rough & destabilizing transition.
[deleted]
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#173Wasn't the answer 42? Also, first question to the new model: "So... any way we could do this with fewer parameters?"
[deleted]
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#174Earlier quoted context omitted.
Was it ever confirmed whether GPT-4 is a Mixture of Experts or not?
I just asked GPT-4, and it denies being a MoE model. User: Are you an MoE model? ChatGPT: No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach. In a GPT model like mine, the entire mod…
Ask it to produce a citation for that claim so we can verify it.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#175Earlier quoted context omitted.
Dozens of millions of dead people is chump change to any empire.
USA hasn't killed more than dozens of millions. If that is chump change then it makes USA looks pretty good for an empire.
Well except that it definitely has. From more than 50 million people in pre-colonial times to less than 3 million in the 90s. USA has done genocide on a scale similar any other ethnic cleansing or to any communist regime.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#176The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…
[deleted]
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#177Earlier quoted context omitted.
Governments have killed more people than any other institution. Corporations have largely made our lives better.
Facebook alone has reeked havoc on at least a few societies. Corporations also enable government actions by developing the tech and tools/weapons needed.
Facebook wreaked havoc, and they reek (stink).
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#178Earlier quoted context omitted.
Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.
The opposite for me. Corporation have naked greed as their driving motivation, but that usually doesn't involve killing off all of their customers. That would be quite unprofitable. People elected to government often seem to seek power for powers sake and I have less faith that they'll not harm us.
OTOH, almost all corporations are less democratic than the most dictatorial governments. Sure, corporations don't (generally) get to be responsible for genocidal armies (except when they do[0]), but "greed" does mean corners get cut on safety, which was a big part of how trade unions in particular (and communism more broadly) got popular in the first place in the latter 1800s and early 1900s.
There's no perfect solution. I think so far we've done best with whatever's the most democratic (which is sometimes trade unions and sometimes capitalism), but even then, democracy itself is what sounds convincing rather than what is true — and the difference between them matters most when you've got an AI trained to respond with what people upvote the most, which is all of them from ChatGPT's RLHF to the Facebook/Twitter/etc. timelines to Google/Amazon/YouTube etc. search results…
[0] TW basically everything https://en.wikipedia.org/wiki/Indian_Rebellion_of_1857#Death...
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#179It will be interesting to see what the government can do here. Can they use their powers to get their hands on the most data? im still skeptical because new techniques are going to give an order of magnitude efficiency boost to transformer models, so 'just waiting' seems like the best approach for now. I dont think they will be able to just skip to the finish line by having the most money.
I just realized that the NSA has probably been able to train GPT-4 equivalents on _all the data_ for a while now. We'll probably never learn about it but that's maybe scarier than just the Snowden collection story because LLMs are so good at retrieval.
Re: Training for one trillion parameter model backed by Intel and US govt has begun
#180A lot of this was new to me, but it looks like Intel hopes to use this to demonstrate the linear scaling capacity of their Aurora nodes. Argonne installs final components of Aurora supercomputer (22 June 2023): https://www.anl.gov/article/argonne-installs-final-component... Aurora Supercomputer Blade Installation Complete (22 June 2023): https://www.intel.com/content/www/us/en/newsroom/news/aurora... Intel® Data Cent…
yes, I can't imagine the architecture of a supercomputer is the right one for LLM training. But maybe? If not, spending years to design and build a system for weather and nuke simulations and ending up doing something that's totally not made for these systems is kind of a mind bender. I can imagine the conversations that led to this: "we need a government owned LLM" "okay what do we need" "lots of compute power" "wel…
How are the optimal architectures for weather simulations and LLM training different?