Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

51–60 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#51

Post by fiedzia »

bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

Really early days of flying where like "let's add a few feathers and then add some more". Though architectural changes were added too.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#53

Post by huijzer »

Majromax wrote: post #41
jonplackett wrote: post #33

Earlier quoted context omitted.

Yeah I'm not totally convinced humans don't have a tremendous amount of training data - interacting with the world for years with constant input from all our senses and parental corrections. I bet if you add up that data it's a lot. But once we are partially trained, training more requires a lot less.

> I bet if you add up that data it's a lot. Let's Fermi estimate that. A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25. Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.…

> Let's Fermi estimate that.

I couldn't agree more. Fermi estimates are underused.

> Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.

And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 50 terabyte [Edited after comment below].

As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization.

[1]: https://youtu.be/kLiwvnr4L80?si=77pXmIBmlp8dsSCG&t=349

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#54

Post by theduder99 »

jojobas wrote: post #30
Pompidou wrote: post #28

Earlier quoted context omitted.

> The first humans were kinda dumb How can you assert this ? Do you have any evidence?

Can't tell if you're joking or serious. In the latter case, the first humans (H. Habilis) had about 1/2 of H. Sapiens brain to work with, and a much smaller fraction of neocortex. If that doesn't satisfy you, let's say I was speaking about some sort of human ancestor before that, which would have been about as dumb as chimps, unless you require proof of their dumbness as well.

Hard to imagine anyone believes these prehuman theories. Just look at the h hablis wiki page to see 90% of it is pure speculation and debated.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#55

Post by ben_w »

yieldcrv wrote: post #52

Mistral 7B parameter models are quite good Already fine tuned and conversational its like education is more important than needing a trillion parameter brainiac

There's a lot we don't know. Human brains appear to be a few hundred trillion parameters, while small rodents are in the realm of tens to hundreds of billions. Would you guess a single sufficiently trained ferret could write on demand short stories about Dracula, Winnie the Poo, and Sherlock teaming up, and follow this up with a bit of university student level web development, and finally give you a decent apple cake recipe? I wouldn't have, and yet the LLMs exist and are much better than I was expecting.

(People who dismiss SotA models as "stochastic parrots" confuse me as much as people who think they're already superhuman; the Markov chains and RNNs I coded a few years back didn't come close to last year's LLMs).

That even smaller models can do well is both unsurprising (why would we expect our existing design efforts to already have the most efficient architecture?) and very surprising (how come we can get something so much less complex than our biological brains to do so much so well?)

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#56

Post by MichaelRazum »

pulse7 wrote: post #5

Isn't GPT-4 already over 1T parameters? And GPT-5 should be even "an order of magnitude" bigger than GPT-4...

Was thinking the same, so 1T might bring you to the league of GPT-4. Actually in the best case, since it seems that meta, google, openai and so on have the most talent.

Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#57

Post by fardinahsan »

vouaobrasil wrote: post #22
skerit wrote: post #20

Earlier quoted context omitted.

Your profile says "I am not rabidly against technology, but I promote a BALANCED, severely critical look at it" "Humanity should give up this useless development of AI" So balanced.

I do not mean balanced against every technology. Some technologies might have some merit. I do believe that AI has no merit, that people developing it are doing is a disservice, and that it should be completely destroyed. OF COURSE, there are some technologies that I think are completely bad, including chemical weapons. In fact, I believe AI to be on the level of chemical weapons essentially.

I don't think saying you have a bone to pick with "AI" is doing your message any favors. What are you specifically against LinearRegression? XGBoost? ResNet? LLMs? AlphaFold?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#58

Post by theropost »

_heimdall wrote: post #44

I'm sure the government's mission is also to develop an AGI that benefits us all.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#59

Post by sixQuarks »

theropost wrote: post #58
_heimdall wrote: post #44

I'm sure the government's mission is also to develop an AGI that benefits us all.

Ideally no one person or entity controls such a thing. But, would I rather have a Government, or a corporation control AGI? If I had to pick one of two evils, the Government would be the lesser of the two.

Has a corporation ever tried to commit genocide?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#60

Post by yieldcrv »

ben_w wrote: post #55
yieldcrv wrote: post #52

Mistral 7B parameter models are quite good Already fine tuned and conversational its like education is more important than needing a trillion parameter brainiac

There's a lot we don't know. Human brains appear to be a few hundred trillion parameters, while small rodents are in the realm of tens to hundreds of billions. Would you guess a single sufficiently trained ferret could write on demand short stories about Dracula, Winnie the Poo, and Sherlock teaming up, and follow this up with a bit of university student level web development, and finally give you a decent apple cake…

maybe ferrets would do better with better interfaces. with better ways to interact with the world and better co-processor. but the main brain might already be capable of those aforementioned things.
Post reply on HN