Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training for one trillion parameter model backed by Intel and US govt has begun

techradar.com

Post reply on HN
Print view View original

201–210 of 267 posts

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#201

Post by bane »

awongh wrote: post #180
bane wrote: post #153

Earlier quoted context omitted.

yes, I can't imagine the architecture of a supercomputer is the right one for LLM training. But maybe? If not, spending years to design and build a system for weather and nuke simulations and ending up doing something that's totally not made for these systems is kind of a mind bender. I can imagine the conversations that led to this: "we need a government owned LLM" "okay what do we need" "lots of compute power" "wel…

How are the optimal architectures for weather simulations and LLM training different?

tbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes a super computer "super"). I believe I read somewhere openAI uses a K8s cluster[1] and other teams I know seem to work with other similar non-unified memory systems. If there's no advantage, part of what makes supercomputers expensive is this memory interconnect, so wouldn't it just be better to use a huge K8s cluster or something?

I'm honestly not sure, and hoping somebody comments here and provides more information as I'm genuinely interested.

1 - https://openai.com/research/scaling-kubernetes-to-7500-nodes

OpenAI says their biggest jobs run on MPI so maybe a supercomputer would be better?

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#202

Post by superhumanuser »

andsoitis wrote: post #107
superhumanuser wrote: post #94

Earlier quoted context omitted.

Governments have killed more people than any other institution. Corporations have largely made our lives better.

Both governments and corporations have generally made our lives better. Both governments and corporations kill people (look up annual tobacco deaths, for just one fun example).

I would not say governments have generally made peoples lives better. I'd wager MOST governments have actually oppressed and depressed their people.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#203

Post by adventured »

jonnycomputer wrote: post #149
dumbfoundded wrote: post #134

Earlier quoted context omitted.

The difference between a government and a corporation is the ability to use violence. A government is just a corporation with a monopoly on violence (police, military, jails...). The structure of how people are organized is more significant. Are we talking about a dictatorship or a functioning democracy? Are we discussing a non-profit or a publicly listed company?

Companies have a long history of using violence, and of local officials looking the other way when they do.

Governments have murdered over one hundred million people over the last few centuries through war and forced famines. Corporations don't even enter that conversation given the scale difference.

Napoleon's government alone murdered more people than all corporations combined have throughout all of history. And that's a revered historical figure that routinely gets fawning movies made about him, there are obviously worse examples. Mao's government murdered several times more than Napoleon did.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#204

Post by Jensson »

charcircuit wrote: post #188

Sadly, I expect this to be a waste of money compared to just using GPT-4. It's hard get to SoA performance.

SoA performance comes from wasting money trying different things and seeing what happens. This will be another data point that we all can learn from, unlike GPT-4 that we have no clue how it works.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#205

Post by CrazyStat »

13of40 wrote: post #167
Majromax wrote: post #41

Earlier quoted context omitted.

> I bet if you add up that data it's a lot. Let's Fermi estimate that. A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25. Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.…

Nit: The bandwidth of your optic nerve is only about 10 kilobits per second. You think you're seeing in 4K but most of it is synthetic.

Academic sources I’ve seen are in the 10Mbps range, e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564115/ (estimating 875kbps for a guinea pig and ~10x that for a human).

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#207

Post by astrange »

mrkeen wrote: post #112
MichaelRazum wrote: post #105

Earlier quoted context omitted.

Have seen an interview with someone from openai. At least they claimed, that they are far away from running out of data and they are able to generate data if needed.

> they are able to generate data if needed. If they can generate data they should use that instead of piping it through a million-dollar lossy compressor.

The point of generating data is so you can teach the model its distribution, not so it can memorize it.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#208

Post by chlorion »

haldujai wrote: post #186
cschmidt wrote: post #160

Earlier quoted context omitted.

I just asked GPT-4, and it denies being a MoE model. User: Are you an MoE model? ChatGPT: No, I am not based on a Mixture of Experts (MoE) model. My underlying architecture is based on the GPT (Generative Pre-trained Transformer) framework, specifically the GPT-4 version. This architecture is a large-scale transformer-based neural network, but it does not use the MoE approach. In a GPT model like mine, the entire mod…

LLMs don’t have insight, these outputs cannot be assumed to be accurate. Due to current hardware limitations it is not feasible to have a 1T parameter model without MoE.

I wonder if it has some canned human-written responses when asking specific questions about itself. This would be pretty clever to silently implement, that will definitely help convince people that it's approaching "AGI".

It's possible that it's just hallucinating here too, I don't have any proof that the responses are canned, but they appear that way to me.

Re: Training for one trillion parameter model backed by Intel and US govt has begun

#210

Post by theptip »

bradley13 wrote: post #6

The solution won't be just "bigger". A model with a trillion parameters will be more expensive to train and to run, but is unlikely to be better. Think of the early days of flight, you had biplanes; then you had triplanes. You could have followed that farther, and added more wings - but it wouldn't have improved things. Improving AI will involve architectural changes. No human requires the amount of training data we…

What? The defining trend of the last 5 or so years is the victory of the scaling hypothesis. More scale = more intelligence. GPT-4 is way smarter than 3.5, this trend is ongoing.

You need more data to utilize more parameters, but folks at the forefront are confident that they are not going to run out of data any time soon.

If you mean “solution to AGI” maybe. But perhaps in-context scratchpads and agent loops will be sufficient to get this architecture to human-level performance, with enough data/parameters. (Sutskever and Hinton have both expressed credulity that the current architectures might get us there.)

All that said, it’s also possible that new architectures will be needed at some point, I’m just pushing back on your claim that we already hit the cliff.

Post reply on HN