Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

developer.nvidia.com

Post reply on HN
Print view View original

31–40 of 100 posts

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#31

Post by alecst »

singularity2001 wrote: post #28
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole.

How much of a simplification is that? And how much does the accuracy of such a model matter, in the grand scheme of things?

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#32

Post by postalrat »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

CPU in kilohertz then megahertz then gigahertz then it stopped.

RAM in kilobytes then megabytes then gigabytes then it stopped.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#33

Post by manquer »

postalrat wrote: post #32
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

CPU in kilohertz then megahertz then gigahertz then it stopped. RAM in kilobytes then megabytes then gigabytes then it stopped.

Those are material science and physical limitations.

Number of parameters in a neural network is not really limited that way, doing useful compute with it is a different matter

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#34

Post by visarga »

alecst wrote: post #31
singularity2001 wrote: post #28

Earlier quoted context omitted.

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…

> All we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is that?

A lot. Parallel optimization is an art form. These models are trained on static datasets, they can't intervene in the environment to infer causal relations, so they need legs and hands.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#35

Post by thewarrior »

petters wrote: post #9

Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.

Maybe that’s what it’s doing under the hood.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#36

Post by shock-value »

dwohnitmok wrote: post #24
posharma wrote: post #18

This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?

Model inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.

Does that hold as the workload scales up? E.g. could this or similar models be used as part of a general-purpose search engine whereby (at least) one inference is completed per unique search? Aside from computation, I know these models consume an intense amount of memory -- would that scale horizontally easily / economically? Would it need to?

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#37

Post by visarga »

bane wrote: post #25

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?

Imagine you lived in a black room and all you can see is a buffer of text scrolling in front of you. Nobody explains what the symbols mean. You don't remember anything medium term, you can only access a short snippet of text at a time and form long term memories gradually. You just look and predict what will come next. Who could become an intelligent person being raised in these conditions?

So they are missing 2 years worth of visual, auditory, tactile and other modalities (grounding), having direct access to change their environment (embodiment) and being part of our society or an AI society (social).

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#38

Post by manquer »

alecst wrote: post #31
singularity2001 wrote: post #28

Earlier quoted context omitted.

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…

I would say quite a bit. Adding even a third body makes it impossible to calculate physics with certainty. A complex system with any number of individual components is hard to understand with certainty and/or calculations can become exponentially more complex .

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#39

Post by gisely »

alecst wrote: post #31
singularity2001 wrote: post #28

Earlier quoted context omitted.

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…

Your professor lied.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#40

Post by manquer »

dwohnitmok wrote: post #24
posharma wrote: post #18

This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?

Model inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.

Not necessarily, you train once , you run inferences billions of times maybe. The compute required could be beyond your resources.
Post reply on HN