Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

developer.nvidia.com

Post reply on HN
Print view View original

91–100 of 100 posts

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#91

Post by YeGoblynQueenne »

cs702 wrote: post #87
YeGoblynQueenne wrote: post #77

Earlier quoted context omitted.

I don't understand this kind of comment. To my mind what it amounts to is "look at how big it is". Alright. So it's big. So what? Is this an elephant pageant? Suppose a friend comes over and says "I went for dinner at a restaurant. Oh my god the portions were sooo big!". Wouldn't you want to know more information about the food and the restaurant, before you decided whether you're interested in it? I appreciate that…

> I don't understand this kind of comment. To my mind what it amounts to is "look at how big it is". Alright. So it's big. So what? It's a good question. A while ago, Rich Sutton wrote a good answer for it.: http://incompleteideas.net/IncIdeas/BitterLesson.html -- I recommend reading the whole essay. Quoting him (emphasis mine): > The biggest lesson that can be read from 70 years of AI research is that general method…

Yes, I've read "The Bitter Lesson". Have you read "A better lesson", by Rodney Brooks?

Edit:

>> A key related question -- to which no one has the answer today -- is whether we must scale computation to match or exceed that of the human brain to be able to replicate or surpass its cognitive abilities.

What "computation" is that? Are you talking about scaling up neural networks, which is more in the context of the conversation, but requires some very big assumptions about (artificial) neural networks? Do you mean a different kind of computation?

(Note: my comment, plus the above edit, is a series of questions and I recognise that commens like that can come across as standoffish. This is not my intention, so please accept the questions above as having been asked in the most neutral tone as possible and in the interest of promoting conversation, rather than confrontation.)

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#92

Post by abecedarius »

TeMPOraL wrote: post #81
abecedarius wrote: post #56

Earlier quoted context omitted.

Emulating a neuron != taking a comparable part in a computation. Probably the former is a lot more complex. For instance, an artificial net can take advantage of backpropagation in a separated training phase -- that's a lot of complexity that's factored out of the runtime phase.

Wonder how this architecture is limiting the space, though - all biological brains train continuously. Our DNNs are more like a brain upload snapshot that's always run for one cycle and then rebooted.

Yeah, that's worth exploring more. It's just that it's not safe to depend on "real neurons are complicated, and therefore artificial nets of simple units won't have transformative capabilities".

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#93

Post by petters »

tgv wrote: post #69
petters wrote: post #9

Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.

A single weight can’t encode an individual word, but the ratio looks close to overfitting too me too.

If 16-bit floating-point numbers are used, it can presumably encode all tokens. In theory. It would not be very easy to work with.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#94

Post by whimsicalism »

YeGoblynQueenne wrote: post #75
whimsicalism wrote: post #68

Earlier quoted context omitted.

https://arxiv.org/pdf/1802.10217.pdf This is the paper I love to link in response to these sort of objections.

Human priors are a feature, not a bug. The reason deep neural networks need so much data to train them (and still do not handle languag nearly as well as humans) is precisely that humans have "background knowledge", things that we already know and don't have to learn all over again from scratch. We bring this knowledge to bear in our ability to learn and understand language. Deep neural networks on the other hand hav…

> Human priors are a feature, not a bug. ... Being forced to learn everything end-to-end, from scratch, is a severe limitation.

They're neither. When performing very human-adjacent tasks, it will certainly put the ML algorithm at a disadvantage compared to us.

But for non-human adjacent tasks, say interpreting what a sequence of amino acids actually means, we can expect the computer to absolutely crush us because our stupid human heuristics take us absolutely nowhere, cause us to see patterns that aren't there, etc. etc.

Regardless, this is irrelevant to the original point that I was making, which is that comparing the performance of DL on human adjacent tasks to the amount of time it takes a human to learn the same task is misguided because you are ignoring the million-year long optimization process to get there.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#95

Post by LuisMondragon »

YeGoblynQueenne wrote: post #77
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

I don't understand this kind of comment. To my mind what it amounts to is "look at how big it is". Alright. So it's big. So what? Is this an elephant pageant? Suppose a friend comes over and says "I went for dinner at a restaurant. Oh my god the portions were sooo big!". Wouldn't you want to know more information about the food and the restaurant, before you decided whether you're interested in it? I appreciate that…

New AI fallacy: appeal to size

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#96

Post by LuisMondragon »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

My issue with this kind of reasoning is the comparison and reference to the human brain. The potential and reach of AI transcends the brain. We never had to master the "mystery" of how birds fly to invent aviation. It was never necessary to compare the number of turbine revolutions of early airplanes to the number of an eagle's feathers. Maybe birds were an inspiration or a metaphor, but thankfully aviation has not been limited to the means of propulsion of the beautiful yet humble pigeon. The potential of aviation has taken us into space exploration and massive international travel. I don't know where AI will take us, but I don't think it will be constrained by this temporary organ called 'human brain'.

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#97

Post by cs702 »

YeGoblynQueenne wrote: post #91
cs702 wrote: post #87

Earlier quoted context omitted.

> I don't understand this kind of comment. To my mind what it amounts to is "look at how big it is". Alright. So it's big. So what? It's a good question. A while ago, Rich Sutton wrote a good answer for it.: http://incompleteideas.net/IncIdeas/BitterLesson.html -- I recommend reading the whole essay. Quoting him (emphasis mine): > The biggest lesson that can be read from 70 years of AI research is that general method…

Yes, I've read "The Bitter Lesson". Have you read "A better lesson", by Rodney Brooks? Edit: >> A key related question -- to which no one has the answer today -- is whether we must scale computation to match or exceed that of the human brain to be able to replicate or surpass its cognitive abilities. What "computation" is that? Are you talking about scaling up neural networks, which is more in the context of the conv…

> Are you talking about scaling up neural networks, which is more in the context of the conversation, but requires some very big assumptions about (artificial) neural networks?

Yes. But note that under the rubric of "deep neural networks" or "deep learning," I would include a lot of things, including combinations of methods like "deep reinforcement learning," learning by self-play via gradual evolution of surviving models, models that use "dense associative memories," of which transformers are only one special case, and future deep learning methods that have not yet been discovered.

And yes, some very big assumptions are required!

FWIW, your comments did not come across as standoffish to me :-)

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#98

Post by jamesbriggs »

bobm_kite9 wrote: post #78

I guess I'm interested to see if this performs qualitatively better than GPT-3, given how many more parameters it has. However, I think this is really a dead-end: throwing more hardware at this is just going to generate better-sounding nonsense. Yes, we are learning the "model" of the English language - which words go with which others, but successively larger transformer models don't really expose much more about th…

Also agree with this, it's almost like a marketing ploy - especially from OpenAI. They produce awesome stuff but things with GPT can get silly sometimes, like when they wouldn't release the larger versions because 'they were too powerful', and in the end you ask it how many eyes my foot has and it says seven...

It producing interesting results, but doesn't really progress the field - although who knows, maybe skynet is actually a 100T parameter transformer

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#99

Post by rajnathani »

cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

If retrieval based NLP [0] becomes a thing, then trillion plus parameters models will likely be less of a thing; as very likely, most of these tens to hundreds of billions of parameters are likely over-fitting (better word: memorized) on training data [text corpus] as seen in the case of GPT-3.

[0] https://ai.stanford.edu/blog/retrieval-based-NLP/

Re: Megatron-Turing NLG 530B, the World’s Largest Generative Language Model

#100

Post by cs702 »

rajnathani wrote: post #99
cs702 wrote: post #5

So we now have models with 0.5 trillion parameters, each the weight of a connection in a neural network. Trillion-parameter models are surely within reach in the near term -- and that's only within two orders of magnitude of the number of synapses in the human brain, which is in the hundreds of trillions, give or take. To paraphrase the popular saying, a trillion here, a trillion there, and pretty soon you're talking…

If retrieval based NLP [0] becomes a thing, then trillion plus parameters models will likely be less of a thing; as very likely, most of these tens to hundreds of billions of parameters are likely over-fitting (better word: memorized) on training data [text corpus] as seen in the case of GPT-3. [0] https://ai.stanford.edu/blog/retrieval-based-NLP/

Yes, self-attention mechanisms are dense associative memories, so it might be possible to replace them in many cases with simpler storage mechanisms. Still, I would count the required storage space as part of a model's parameter size -- e.g., a model consisting of 1 trillion values in RAM and 99 trillion values in storage consists of... 100 trillion values.
Post reply on HN