Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

41–50 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#41

Post by Majromax »

jonplackett wrote: post #33
elihu wrote: post #23

Earlier quoted context omitted.

> "No human requires the amount of training data we are already giving the models." Well, humans are also trained differently. We interact with other humans in real time and get immediate feedback on their responses. We don't just learn by reading through reams of static information. We talk to people. We get into arguments. And so on. Maybe the ideal way to train an AI is to have it interact with lots of humans, so…

Yeah I'm not totally convinced humans don't have a tremendous amount of training data - interacting with the world for years with constant input from all our senses and parental corrections. I bet if you add up that data it's a lot. But once we are partially trained, training more requires a lot less.

> I bet if you add up that data it's a lot.

Let's Fermi estimate that.

A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25.

Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.

There's plenty of reason to think that we don't learn effectively from all of this data (there's lots of redundancy, for example), but at the grossest orders of magnitude you seem to be right that at least in theory we have access to a tremendous amount of data.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#42

Post by ArekDymalski »

vouaobrasil wrote: post #11
danielbln wrote: post #8

Earlier quoted context omitted.

It might be dangerous, but how is it useless? There is also an enormous positive gain to be had in terms of autonomous discovery, science progress and so on. And prisoner's dilemma means "we" will not stop pursuing this, even of there is a net negative benefit to humanity overall.

Yes, I believe there is a net negative benefit. I do not think there is an enormous positive gain. Science progress in my opinion has gone too far in many respects and we should not be able to generate science so easily because we will not have the wisdom to use it. Although I love science, it's use is far too lackadaisical now -- see for example the climate crisis and efficient resource extraction of fossil fuels. T…

> Science progress in my opinion has gone too far in many respects and we should not be able to generate science so easily because we will not have the wisdom to use it.

I think it was always the case. The wisdom actually comes from mistakes too, not from a priori speculations. Especially as all new solutions/technologies have been criticized and judged as dangerous, frivolous, unnecessary, morally wrong etc. Bikes, cars, street lights,social equality and so long.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#43

Post by jsnell »

vouaobrasil wrote: post #17
omneity wrote: post #14

Earlier quoted context omitted.

What makes you think this technology is out of control? If anything a 1T param model would only run in like 5-10 computers in this world. As much control as it gets. By the way this is doomer rhetoric. Bashing scientific advancements as dangerous or useless by trying to attach vacuously scary ideas that are as speculative as they are outlandish.

It's hardly vacuous. What about these scientific advancements? 1. Fossil fuel extraction -- we are killing our planet. 2. Plastic chemistry -- there are 5 trillion pieces of plastic in the ocean killing seabirds, turtles, fish ( https://doi.org/10.1371/journal.pone.0111913 ) 3. Concrete engineering -- building cities that move people away from a connection with nature 4. Veterinary medicine -- supporting global meat…

None of that has anything to do with your original complaint of AI models. It makes it look like you don't actually have any argument against AI development, but it is just another thing to rail against.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#45

Post by junofan »

Zanni wrote: post #31
jojobas wrote: post #30

Earlier quoted context omitted.

Can't tell if you're joking or serious. In the latter case, the first humans (H. Habilis) had about 1/2 of H. Sapiens brain to work with, and a much smaller fraction of neocortex. If that doesn't satisfy you, let's say I was speaking about some sort of human ancestor before that, which would have been about as dumb as chimps, unless you require proof of their dumbness as well.

Not your original objector, but ... I was willing to accept that early humans are dumb, but your explanation relies on evolution and not social interaction.

We know their brains didn’t grow much after birth, unlike humans, which also suggests faster maturity akin to living apes and likely less social support for extended adolescence.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#46

Post by Aardwolf »

bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

Unlike biplanes, CPU's with more transistors are more powerful than those with less. And adding more CPU cores keeps increasing the amount of threads you can run at the same time.

Why would LMM's be more like the biplanes analogy, and less like the CPU analogy?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#47

Post by vouaobrasil »

jsnell wrote: post #43
vouaobrasil wrote: post #17

Earlier quoted context omitted.

It's hardly vacuous. What about these scientific advancements? 1. Fossil fuel extraction -- we are killing our planet. 2. Plastic chemistry -- there are 5 trillion pieces of plastic in the ocean killing seabirds, turtles, fish ( https://doi.org/10.1371/journal.pone.0111913 ) 3. Concrete engineering -- building cities that move people away from a connection with nature 4. Veterinary medicine -- supporting global meat…

None of that has anything to do with your original complaint of AI models. It makes it look like you don't actually have any argument against AI development, but it is just another thing to rail against.

I have many specific arguments against AI:

1. It will take away jobs from creatives

2. It will reduce the need for other human beings

3. It will be used by criminals for identity theft...

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#48

Post by scriptsmith »

huijzer wrote: post #12
bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

Karpathy in his recent video [1] agrees, but at this point scaling is a very reliable way to better accuracy. [1]: https://youtu.be/zjkBMFhNj_g?si=eCH04466rmgBkHDA

Seems like he actually disagrees here:

If you train a bigger model on more text, we have a lot of confidence that the next-word prediction task will improve. So algorithmic progress is not necessary, it's a very nice bonus, but we can sort of get more powerful models for free, because we can just get a bigger computer, which we can say with some confidence we're going to get, and just train a bigger model for longer, and we are very confident we are going to get a better result.

https://youtu.be/zjkBMFhNj_g?t=1543 (23:43)

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#49

Post by wongarsu »

Aardwolf wrote: post #46
bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

Unlike biplanes, CPU's with more transistors are more powerful than those with less. And adding more CPU cores keeps increasing the amount of threads you can run at the same time. Why would LMM's be more like the biplanes analogy, and less like the CPU analogy?

In general you can view "understanding" as a compression of information. You take in a bunch of information, detect an underlying pattern, and remember the pattern and necessary context, instead of the entire input.

The "problem" with larger neural networks is that they can store more information, so they can substitute understanding with memorization. Something similar happens with human students, who can stuff lots of raw information into short-term-memory, but to fit it into the much more precious long-term-memory you have to "understand" the topic, not just memorize it. In neural networks we call that memorization a failure to generalize. Just like a human, a network that just memorizes doesn't do well if you ask it about anything slightly different than the training data.

Of course it's a balance act, because a network that's too small doesn't have space to store enough "understanding" and world model. A lot of the original premise of OpenAI was to figure out if LLMs keep getting better if you make them bigger, and so far that has worked. But there is bound to be a ceiling on this, where making the model bigger starts making it dumber.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#50

Post by hugh-avherald »

_heimdall wrote: post #44

I'm sure the government's mission is also to develop an AGI that benefits us all.

I am not too keen on the US Government being in command of an AGI, but there is only one other entity capable of developing an AGI before the US Government. And I'm less keen on it being the one to control it.
Post reply on HN