Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

71–80 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#71

Post by [deleted] »

mcosta wrote: post #38
vouaobrasil wrote: post #17

Earlier quoted context omitted.

It's hardly vacuous. What about these scientific advancements? 1. Fossil fuel extraction -- we are killing our planet. 2. Plastic chemistry -- there are 5 trillion pieces of plastic in the ocean killing seabirds, turtles, fish ( https://doi.org/10.1371/journal.pone.0111913 ) 3. Concrete engineering -- building cities that move people away from a connection with nature 4. Veterinary medicine -- supporting global meat…

Well, I don't care about nonhuman animals. And being sincere, I don't care about 99% of human animals. I cry for the deaths of my loved ones, and forget the rest. It is the only way to remain sane.

[deleted]

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#72

Post by isolli »

jefftk wrote: post #67
bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

> you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. But people did try! https://en.wikipedia.org/wiki/Besson_H-5 https://en.wikipedia.org/wiki/Multiplane_(aeronautics)#Quadr...

Why stop here? This engineer experimented with 20 wings and more:

https://en.wikipedia.org/wiki/Horatio_Frederick_Phillips

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#73

Post by [deleted] »

mcosta wrote: post #38
vouaobrasil wrote: post #17

Earlier quoted context omitted.

It's hardly vacuous. What about these scientific advancements? 1. Fossil fuel extraction -- we are killing our planet. 2. Plastic chemistry -- there are 5 trillion pieces of plastic in the ocean killing seabirds, turtles, fish ( https://doi.org/10.1371/journal.pone.0111913 ) 3. Concrete engineering -- building cities that move people away from a connection with nature 4. Veterinary medicine -- supporting global meat…

Well, I don't care about nonhuman animals. And being sincere, I don't care about 99% of human animals. I cry for the deaths of my loved ones, and forget the rest. It is the only way to remain sane.

[deleted]

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#74

Post by pajko »

sixQuarks wrote: post #59
theropost wrote: post #58

Earlier quoted context omitted.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

Think Weyland-Yutani.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#75

Post by hliyan »

elihu wrote: post #23
bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

> "No human requires the amount of training data we are already giving the models." Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on. Maybe the ideal way to train an AI is to have it interact with lots of humans, so…

Also, humans have multiple internal models for various aspects of reality, that seem to exist as a layer on top of the "raw" training. They have the ability to extrapolate based on those models (which transformers cannot do, as far I as understand). Perhaps GP is right -- perhaps what's missing in the architecture is an analog of these models.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#76

Post by huijzer »

scriptsmith wrote: post #48
huijzer wrote: post #12

Earlier quoted context omitted.

Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy. [1]: https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgBkHDA

Seems like he actually disagrees here: If you train a bigger model on more text, we have a lot of confidence that the next-word prediction task will improve. So algorithmic progress is not necessary, it's a very nice bonus, but we can sort of get more powerful models for free, because we can just get a bigger computer, which we can say with some confidence we're going to get, and just train a bigger model for longer,…

And then at 35 minutes he spends a few minutes talking about ideas for algorithmic improvements.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#77

Post by checkyoursudo »

sixQuarks wrote: post #59
theropost wrote: post #58

Earlier quoted context omitted.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

A more pressing reformulation might be, has a corporation ever explicitly tried not to commit genocide?

Google, maybe, until they dropped the "don't be evil" catchphrase?

Many, maybe even most, governments could commit genocide and don't.

How many corporations that could commit genocide wouldn't? I don't know the answer to this, but I am less confident about the answer than I would prefer.

I don't think most corporations are inherently more evil than most governments, but I would choose a democratic government or consortium to shepherd AGI over any corporation, but also of course over any non-democratic government as well.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#78

Post by alfiedotwtf »

vouaobrasil wrote: post #47
jsnell wrote: post #43

Earlier quoted context omitted.

None of that has anything to do with your original complaint of AI models. It makes it look like you don't actually have any argument against AI development, but it is just another thing to rail against.

I have many specific arguments against AI: 1. It will take away jobs from creatives 2. It will reduce the need for other human beings 3. It will be used by criminals for identity theft...

> 1. It will take away jobs from creatives

"In the 17th century nearly all Europe experienced revolts of the workpeople against the ribbonloom, a machine for weaving ribbons and trimmings, called in Germany Bandmühle, Schnurmühle, and Mühlenstuhl. These machines were invented in Germany. Abbé Lancellotti, in a work that appeared in Venice in 1636, but which was written in 1579, says as follows:

Anthony Müller of Danzig saw about 50 years ago in that town, a very ingenious machine, which weaves 4 to 6 pieces at once. But the Mayor being apprehensive that this invention might throw a large number of workmen on the streets, caused the inventor to be secretly strangled or drowned"

-- "Section 5: The Strife Between Workman and Machine", Das Kapital

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#79

Post by Denote6737 »

dmix wrote: post #68
steve1977 wrote: post #35

Earlier quoted context omitted.

"Sure, just quickly give me unrestricted access to the system" "Ok. Well, thinking about it, maybe that's not such a good idea safety-wise, I think you'll have to give back that access" "I'm sorry, Dave. I'm afraid I can't do that."

Fun fact, in the sequel 2010 you learn that Hal didn't really go rogue like an AGI, it was following preprogrammed conditions set by the US government which put the mission at higher priority than the crew, changing some parameters without telling the mission designers, which put them at risk. So it was technically just following orders in the cold way a machine does.

The wonderful thing about computers is that they do exactly what you tell them to. The terrible thing about computers is that they do exactly what you tell them to.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#80

Post by checkyoursudo »

dmix wrote: post #68
steve1977 wrote: post #35

Earlier quoted context omitted.

"Sure, just quickly give me unrestricted access to the system" "Ok. Well, thinking about it, maybe that's not such a good idea safety-wise, I think you'll have to give back that access" "I'm sorry, Dave. I'm afraid I can't do that."

Fun fact, in the sequel 2010 you learn that Hal didn't really go rogue like an AGI, it was following preprogrammed conditions set by the US government which put the mission at higher priority than the crew, changing some parameters without telling the mission designers, which put them at risk. So it was technically just following orders in the cold way a machine does.

> didn't really go rogue like an AGI

Except, that might really be how an AGI eventually goes rogue in the first place! But, no, I didn't know that. It is a fun fact indeed.

Post reply on HN