Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

developer.nvidia.com

Post reply on HN
Print view View original

21–30 of 100 posts

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#21

Post by sanity31415 »

buffington wrote: post #20
posharma wrote: post #18

This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?

When you say "inference", do you mean "interface", or is "inference" an ML term I'm not familiar with?

First you train a model then you use it, "inference" is a fancy word for using the model.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#22

Post by igorkraw »

buffington wrote: post #20
posharma wrote: post #18

This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?

When you say "inference", do you mean "interface", or is "inference" an ML term I'm not familiar with?

It's a ML term, inference basically means using the probability model you learned to draw "inferences" about a piece of data. In this context, it means giving the language model some context and using some method (either arg max sampling or something more sophisticated like beam search) to do what amounts to statistical auto implemention on it. As you might imagine, doing this with 530 GB of data at speed is quite energy intensive, even though there are things you can do to compress the model (distillation, pruning, compression/discretization) and specialised inference hardware.

Technically there is some very specific meaning to inference vs. prediction, but it's been heavily overloaded with meaning by now

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#23

Post by lucidrains »

knuthsat wrote: post #19
petters wrote: post #9

Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.

For some reason this issue with model having insane amounts of weights but training data being small is not something that is an issue for modern NNs.

https://arxiv.org/abs/2109.02355

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#24

Post by dwohnitmok »

posharma wrote: post #18

This is great. Now, how do we inference these models economically? It appears there's some kind of competition to train larger and larger models, but the inferencing side of the story seems to be neglected?

Model inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#25

Post by bane »

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#26

Post by dlkf »

bane wrote: post #25

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?

Humans use language to accomplish tasks in their environment - establishing relationships, making deals, coaxing others, etc. By contrast, all neural language models do is predict the next word as a function of the previous word. So far, these language models have nothing at all to do with language learning. They're only valuable insofar as they advance downstream engineering tasks like machine translation.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#28

Post by singularity2001 »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own.

Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#30

Post by rolisz »

ImprobableTruth wrote: post #15
fspeech wrote: post #14

Earlier quoted context omitted.

A 10 trillion parameter model was mentioned here: https://mobile.twitter.com/ethancaballero/status/14458268620...

That's MoE.

Mixture of Experts, aka not all 10 trillion parameters are used at the same time, just a subset that is an "expert" on the "task at hand".
Post reply on HN