Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

61–70 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#61

Post by wodenokoto »

bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

OpenAIs big innovation was “bigger”. It is not clear when we should stop scaling.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#62

Post by Majromax »

huijzer wrote: post #53
Majromax wrote: post #41

Earlier quoted context omitted.

> I bet if you add up that data it's a lot. Let's Fermi estimate that. A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25. Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.…

> Let's Fermi estimate that. I couldn't agree more. Fermi estimates are underused. > Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives abo…

> Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte.

Thanks, I was looking for that estimate but could not quickly find it. (Although are you sure about that 'petabyte' figure? I see 1e13 * 5 = 50e13 = 50 TB?)

I love these sorts of bulk comparisons, since they let us start to reason about how much humans learn from "each bit of experience" versus LLMs. On one hand humans process a lot of sensory information (most of it by discarding, mind), but on the other humans do a lot of 'active learning' whereby experience is directly related to actions. LLMs don't have the latter, since they're trained on passive feeds of tokens.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#63

Post by bradchris »

sixQuarks wrote: post #59
theropost wrote: post #58

Earlier quoted context omitted.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

Well, there’s the entire history of Dole and also the United Fruit Company in South America.

In more recent history, Exxon has a checkered past when it comes to using force against sovereign citizens, often employing paramilitary organizations to guard oil fields [1] to more recently abusing international court systems to disbar and jail the lawyer which successfully secured judgements against them [2]

And at this very moment, right now, is the ongoing genocide in the Congo due to rare earth mineral mining [3]

Everything I link here is quite frankly the tip of the iceberg, not the end-all-be-all, but meant to provide a jumping off point should you want to research how often private capital and genocide go hand in hand.

[1] https://en.m.wikipedia.org/wiki/Accusations_of_ExxonMobil_hu...

[2] https://amp.theguardian.com/us-news/2021/mar/28/chevron-lawy...

[3] https://republic.com.ng/october-november-2023/congo-cobalt-g...

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#64

Post by CBarkleyU »

sixQuarks wrote: post #59
theropost wrote: post #58

Earlier quoted context omitted.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

The Dutch East India Company definitely did things that are well in the genocide category, the British East India company did things that - while not on that level of evil - can only be described as ruthlessly atrocious.

Pretty much all leading heads of IG Farben were in one way or another dinked (but not procecuted bc of political reasons) for their part in the Holocaust.

Then we have the Kiobel vs Royal Dutch Petroleum Co. case, where a number of petroleum companies ("allegedly") instigated the Nigarian government to crush protests.

It all depends on your definition of "trying to commit [atrocity]", to me personally, instigating and happily lending a hand are both morally deplorable. So in the context of this thread, I'd much rather have the instigater have that power than a helper. At least I have some say in what the government does... I hope?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#65

Post by jefftk »

theropost wrote: post #58
_heimdall wrote: post #44

I'm sure the government's mission is also to develop an AGI that benefits us all.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Except that it's a big open question how to control an AGI. So the choice may not be between which entity controls it, but whether it is created by an entity that is able to control it at all.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#66

Post by washadjeffmad »

A lot of this was new to me, but it looks like Intel hopes to use this to demonstrate the linear scaling capacity of their Aurora nodes.

Argonne installs final components of Aurora supercomputer (22 June 2023): https://www.anl.gov/article/argonne-installs-final-component...

Aurora Supercomputer Blade Installation Complete (22 June 2023): https://www.intel.com/content/www/us/en/newsroom/news/aurora...

Intel® Data Center GPU Max Series, previously codename Ponte Vecchio (31 May 2023): https://www.intel.com/content/www/us/en/developer/articles/t...

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#67

Post by jefftk »

bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

> you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things.

But people did try! https://en.wikipedia.org/wiki/Besson_H-5 https://en.wikipedia.org/wiki/Multiplane_(aeronautics)#Quadr...

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#68

Post by dmix »

steve1977 wrote: post #35
ajdegol wrote: post #32

Wasn't the answer 42? Also, first question to the new model: "So... any way we could do this with fewer parameters?"

"Sure, just quickly give me unrestricted access to the system" "Ok. Well, thinking about it, maybe that's not such a good idea safety-wise, I think you'll have to give back that access" "I'm sorry, Dave. I'm afraid I can't do that."

Fun fact, in the sequel 2010 you learn that Hal didn't really go rogue like an AGI, it was following preprogrammed conditions set by the US government which put the mission at higher priority than the crew, changing some parameters without telling the mission designers, which put them at risk. So it was technically just following orders in the cold way a machine does.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#69

Post by btbuildem »

sixQuarks wrote: post #59
theropost wrote: post #58

Earlier quoted context omitted.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

The first chartered corps were created in order to streamline genocide - East India for example. This new form combined the merchants' wealth and the royalty's military power into an entity that was accountable to no one but those who stood to profit from it.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#70

Post by meroes »

sixQuarks wrote: post #59
theropost wrote: post #58

Earlier quoted context omitted.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

Yes? United Fruit Company now known as Chiquita.
Post reply on HN