Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s New Language Model, Chinchilla

marktechpost.com

Post reply on HN
Print view View original

51–60 of 145 posts

Re: DeepMind’s New Language Model, Chinchilla

#51

Post by eutectic »

adamsmith143 wrote: post #39
eigenvalue wrote: post #25

Earlier quoted context omitted.

According to the LessWrong post, the smaller model trained on more data performs better on most of the tasks, but it’s worse on “college level math” questions. I wonder why that is. Is it because the extra capacity of the larger model was used to basically memorize theorems? Or is it because the extra “brain power” let it model the math better? Oddly, one of the tasks that the smaller most outperformed the larger mod…

Probably right. Most people dump on these language models for this reason but it would be absurd for a HS student to have to re-derive the quadratic equation every time they worked on an Algebra problem so naturally you memorize it. Why should it be any different for a language model?

I never memorized the quadratic formula, and I did OK.

Re: DeepMind’s New Language Model, Chinchilla

#52

Post by Delitio »

sudosysgen wrote: post #41
VikingCoder wrote: post #33

Earlier quoted context omitted.

70 billion parameters... Is each of those a 4-byte float? So, is that 280 billion bytes of just parameters?

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Is there any source which explains what billion of parameters actually are?

In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender.

Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, height,

Re: DeepMind’s New Language Model, Chinchilla

#53

Post by axg11 »

adamsmith143 wrote: post #42
mrfusion wrote: post #20

Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

Some estimates:

- 500M tweets per day

- 30 words/tokens per tweet

- 40% of all tweets thrown away due to being duplicate/spam/bots

= 9B tokens generated per day

Re: DeepMind’s New Language Model, Chinchilla

#54

Post by jefft255 »

Delitio wrote: post #52
sudosysgen wrote: post #41

Earlier quoted context omitted.

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, h…

The parameters are the number of weights in a neural network, in this case.

Re: DeepMind’s New Language Model, Chinchilla

#55

Post by sillysaurusx »

ShamelessC wrote: post #40
eigenvalue wrote: post #25

Earlier quoted context omitted.

According to the LessWrong post, the smaller model trained on more data performs better on most of the tasks, but it’s worse on “college level math” questions. I wonder why that is. Is it because the extra capacity of the larger model was used to basically memorize theorems? Or is it because the extra “brain power” let it model the math better? Oddly, one of the tasks that the smaller most outperformed the larger mod…

Gwern responded to a similar question in the comments section. (parent) > the fact that data and compute need to scale proportionally seems… like a big point in favor of NNs as memorizers/interpolators. (gwern) > Surely it's the opposite? The more bang you get out of each parameter, the less it looks like 'just' (whatever that means) memorization/interpolation. When you needed to increase parameters a lot, disproport…

This isn’t addressing their question. And Gwern’s goal here is to (incorrectly) try to get rid of the idea that models are just memorizing and interpolating, when in fact memorization and interpolation is what we all do, including models. He’s just bothered by the idea that people think of models as less than magic.

On the other hand, https://twitter.com/model_mechanic/status/151297688118364569... is admittedly pretty magical, even if the basis of that magic is memorization and interpolation.

Re: DeepMind’s New Language Model, Chinchilla

#56

Post by matt123456789 »

Delitio wrote: post #52
sudosysgen wrote: post #41

Earlier quoted context omitted.

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, h…

A parameter is a scalar value, most of which are in the attention matrices and feedforward matrices, you also hear these called “weights”. Any intro to DL course will cover these in detail. I recommend started with Andrew Ng’s Coursera class on Intro to Machine Learning, although there may be better ones out there now.

Re: DeepMind’s New Language Model, Chinchilla

#57

Post by ianbutler »

adamsmith143 wrote: post #37
ianbutler wrote: post #7

This is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the ability for smaller independent groups to train and release their own versions, more fully democratizing AI. Looking forward to a group like Eluther or Hugging Face releasing a version of this.

>This is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the ability for smaller independent groups to train and release their own versions, more fully democratizing AI. I think I support this in principle but it seems like the scaling curves keep going so it's easier to just make larger models with more data. >Looking forw…

Small is relative -- and to Google, Facebook and Microsoft they're positively tiny. Perfect is the enemy of good or some such and I think this is a move in the right direction even if I can't personally train this on my 3090.

Re: DeepMind’s New Language Model, Chinchilla

#58

Post by dotnet00 »

Delitio wrote: post #52
sudosysgen wrote: post #41

Earlier quoted context omitted.

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, h…

Parameters are just floating point numbers, at most they can be seen as degrees of freedom or kind of like the order of a polynomial used in curve fitting.

They're too abstract to assign much meaning to individual parameters, as our understanding of why their values are exactly the way they are is extremely limited.

Re: DeepMind’s New Language Model, Chinchilla

#59

Post by jankeymeulen »

MisterTea wrote: post #30
binarymax wrote: post #9

Earlier quoted context omitted.

Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.

My theory is since no one reads literature anymore, timeless, interesting and unique names from history and other cultures are lost to a deluge of soon to be forgotten gag, pop-culture and meme names. Perhaps this is why we have Chinchilla and not Oberon.

Like the Oberon OS and programming language?

Re: DeepMind’s New Language Model, Chinchilla

#60

Post by zarzavat »

adamsmith143 wrote: post #42
mrfusion wrote: post #20

Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully", would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ?

It's clear that these models have orders of magnitude too much data already.

It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.

Post reply on HN