Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

developer.nvidia.com

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#51

Post by google234123 »

manquer wrote: post #38
alecst wrote: post #31

Earlier quoted context omitted.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…

I would say quite a bit. Adding even a third body makes it impossible to calculate physics with certainty. A complex system with any number of individual components is hard to understand with certainty and/or calculations can become exponentially more complex .

Citing the fact that the 3 body problem doesn't always have an exact solution is a straw man argument.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#52

Post by ZoomerCretin »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

China's Wu Dao 2.0 has 1.75 trillion parameters. https://towardsdatascience.com/gpt-3-scared-you-meet-wu-dao-...

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#53

Post by dustfinger »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

The switch transformer has already achieved a trillion parameters.

https://arxiv.org/abs/2101.03961

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#54

Post by jowday »

sonic-boom wrote: post #47

Any idea if they’ll release an API similar to GPT-3? It’s great that larger and larger models are trained but without enabling access to the trained models developers are left out from the progress…

I hope they don’t release an API the way they released an API for GPT-3.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#55

Post by manquer »

google234123 wrote: post #51
manquer wrote: post #38

Earlier quoted context omitted.

I would say quite a bit. Adding even a third body makes it impossible to calculate physics with certainty. A complex system with any number of individual components is hard to understand with certainty and/or calculations can become exponentially more complex .

Citing the fact that the 3 body problem doesn't always have an exact solution is a straw man argument.

The intent was to illustrate that complexities of a system of simple components can be pretty difficult. Automata theory has more appropriate examples perhaps.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#56

Post by abecedarius »

singularity2001 wrote: post #28
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Emulating a neuron != taking a comparable part in a computation. Probably the former is a lot more complex. For instance, an artificial net can take advantage of backpropagation in a separated training phase -- that's a lot of complexity that's factored out of the runtime phase.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#57

Post by andyxor »

alecst wrote: post #31
singularity2001 wrote: post #28

Earlier quoted context omitted.

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…

That’s correct understanding as of 1943 when the “artificial neural network“ model your professor is teaching was developed.

There is a whole lot of new knowledge on how live neurons and networks of neurons work that had been collected in the last 75 years in the neuroscience domain but it’s mostly ignored by computer scientists.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#60

Post by sanxiyn »

postalrat wrote: post #32
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

CPU in kilohertz then megahertz then gigahertz then it stopped. RAM in kilobytes then megabytes then gigabytes then it stopped.

Yes for CPU, no for RAM. You can buy a computer with terabytes of RAM just fine. It's just expensive.
Post reply on HN