Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

developer.nvidia.com

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#71

Post by tudorw »

singularity2001 wrote: post #28
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

Except that every synapse is not a dumb weight but a highly complex system connected to an even more complex system (aka neuron) which might each be a (super)computer on its own. Given how extremely bad we are at computing, there is hope (for ai) that the neurons or their circuits are not _that_ powerful after all.

Won't somebody think of the exosomes and telocytes?

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#72

Post by inciampati »

tgv wrote: post #69
petters wrote: post #9

Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.

A single weight can’t encode an individual word, but the ratio looks close to overfitting too me too.

I've often wondered if a lighter reinforcement learning based model on top of a full text index might do as well or better than these putatively overfit language models. Curious if anyone knows of ongoing or recent work on this approach.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#73

Post by qeternity »

andyxor wrote: post #57
alecst wrote: post #31

Earlier quoted context omitted.

Last I heard (and I believe this could be wrong) my professor said that we basically understand how a single neuron works. That like basically if we do X input we get Y output, up to some accuracy. He used this to discuss the idea behind neural networks -- that each neuron is simple enough to model, all we need to worry about is the weights and the dynamics of the network as a whole. How much of a simplification is t…

That’s correct understanding as of 1943 when the “artificial neural network“ model your professor is teaching was developed. There is a whole lot of new knowledge on how live neurons and networks of neurons work that had been collected in the last 75 years in the neuroscience domain but it’s mostly ignored by computer scientists.

I’d be really interested in learning more about this. Can you point me to some easily grokable literature?

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#74

Post by niklasd »

shock-value wrote: post #36
dwohnitmok wrote: post #24

Earlier quoted context omitted.

Model inference is actually comparatively very cheap. If you have the resources to train a model, you most definitely have the resources to run it.

Does that hold as the workload scales up? E.g. could this or similar models be used as part of a general-purpose search engine whereby (at least) one inference is completed per unique search? Aside from computation, I know these models consume an intense amount of memory -- would that scale horizontally easily / economically? Would it need to?

Google is using BERT for most/all search queries [1]. BERT is far smaller than Megatron (340M [1] https://blog.google/products/search/search-language-understa...

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#75

Post by YeGoblynQueenne »

whimsicalism wrote: post #68
bane wrote: post #25

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?

https://arxiv.org/pdf/1802.10217.pdf This is the paper I love to link in response to these sort of objections.

Human priors are a feature, not a bug. The reason deep neural networks need so much data to train them (and still do not handle languag nearly as well as humans) is precisely that humans have "background knowledge", things that we already know and don't have to learn all over again from scratch. We bring this knowledge to bear in our ability to learn and understand language. Deep neural networks on the other hand have only a very, very limited ability to represent and therefore use background knowledge- and so they have to make up for it with astounding amounts of data.

The OP's criticism is valid. Being forced to learn everything end-to-end, from scratch, is a severe limitation.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#76

Post by sytelus »

bane wrote: post #25

What's really interesting is that these models are using some non-trivial portion of all easily accessible human writing -- yet humans learn language really well with significantly less input data. What's missing in the field to replicate human performance in learning?

We sure don't read entire wikipedia but out speech/text consumption is pretty high and we take long time to learn. At 150 words/minute, I would say babies probably consume about 10 million tokens before they start to speak. Baby's training time is much higher than just few days compared for a GPU cluster. Also, baby's vocab is very small and can do very limited things compared to these large models (for example, can't answer who is the king of England or what is the capital of Bangladesh).

This is not to say that language models are efficient, of course. That's not even remotely true. But we seem to under-estimate how much time and resources we need to learn something.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#77

Post by YeGoblynQueenne »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

I don't understand this kind of comment. To my mind what it amounts to is "look at how big it is". Alright. So it's big. So what? Is this an elephant pageant?

Suppose a friend comes over and says "I went for dinner at a restaurant. Oh my god the portions were sooo big!". Wouldn't you want to know more information about the food and the restaurant, before you decided whether you're interested in it?

I appreciate that "big" is in peoples' minds associated with "strong", but most of the work in making language models bigger and bigger goes against the normal trend in computer science [1] and also neural networks research in gneral where the trend is to constantly try to reduce the size of models and improve their data efficiency.

What's worse, the trend to supersize language models is never justified, either theoretically (ha ha) or empirically in the relevant literature - and when rival teams make the obvious experiments the evidence is that size is not required to achieve good performance. For example:

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

https://aclanthology.org/2021.naacl-main.185/

____________

[1] Imagine someone bragging that their mergesort implementation has a million LOC! People brag about implementations in few lines of code, not many.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#78

Post by bobm_kite9 »

I guess I'm interested to see if this performs qualitatively better than GPT-3, given how many more parameters it has.

However, I think this is really a dead-end: throwing more hardware at this is just going to generate better-sounding nonsense. Yes, we are learning the "model" of the English language - which words go with which others, but successively larger transformer models don't really expose much more about the nature of intelligent conversation.

I think we need a better algorithm now.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#79

Post by simonh »

bobm_kite9 wrote: post #78

I guess I'm interested to see if this performs qualitatively better than GPT-3, given how many more parameters it has. However, I think this is really a dead-end: throwing more hardware at this is just going to generate better-sounding nonsense. Yes, we are learning the "model" of the English language - which words go with which others, but successively larger transformer models don't really expose much more about th…

Listening to conversations and learning how they flow is only one aspect of language learning. What these things are missing is the interactive part. Next generation systems need to be able to form hypotheses about what appropriate responses should be, try out various responses and then see what the results are. They can only learn so much from passive consumption of training sets, so I agree this approach is going to hit a wall of diminishing returns.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#80

Post by TeMPOraL »

YeGoblynQueenne wrote: post #77
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

I don't understand this kind of comment. To my mind what it amounts to is "look at how big it is". Alright. So it's big. So what? Is this an elephant pageant? Suppose a friend comes over and says "I went for dinner at a restaurant. Oh my god the portions were sooo big!". Wouldn't you want to know more information about the food and the restaurant, before you decided whether you're interested in it? I appreciate that…

> I appreciate that "big" is in peoples' minds associated with "strong"

In my mind, this is now called Pakled reasoning, from the scene in Star Trek: Lower Decks.

  Pakled rebel turned leader:
  "I am now Pakled leader. Behold my giant helmet!"

  Other Pakled:
  "He is strong!"
https://www.youtube.com/watch?v=lv1uhAa_M_U&t=193s
Post reply on HN