Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s New Language Model, Chinchilla

marktechpost.com

Post reply on HN
Print view View original

41–50 of 145 posts

Re: DeepMind’s New Language Model, Chinchilla

#41

Post by sudosysgen »

VikingCoder wrote: post #33
ShamelessC wrote: post #5

Seems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin... > On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute. > Following the new scaling la…

70 billion parameters... Is each of those a 4-byte float? So, is that 280 billion bytes of just parameters?

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Re: DeepMind’s New Language Model, Chinchilla

#42

Post by adamsmith143 »

mrfusion wrote: post #20

Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

Re: DeepMind’s New Language Model, Chinchilla

#43

Post by jstx1 »

binarymax wrote: post #9
galcerte wrote: post #3

I have to ask, why call it that? I had a chuckle once I saw the name.

Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.

Image models too - the Inception paper from 2014 directly refers to knowyourmeme.com and the "we need to go deeper" meme from the movie Inception - https://knowyourmeme.com/memes/we-need-to-go-deeper - it's the first reference in the paper [1] and it's also why the model is called that way.

[1] https://arxiv.org/pdf/1409.4842.pdf

Re: DeepMind’s New Language Model, Chinchilla

#44

Post by screye »

sirk390 wrote: post #34

Is outperforming GPT-3 still a good reference? It seems there are many models outperforming GPT-3 in the superglue benchmark: https://super.gluebenchmark.com/leaderboard/ GPT-3 is in position #21, with 71.8% score. The best model is at 91.2%. Note the human baseline in #6 with 89.8%

Note that this isn't an apples-to-apples comparison. The GPT-3 position is for a few-shot use-case that has not been trained for this particular task. When fine-tuned, GPT-3 would be expected to perform a lot better. Lastly, GPT-3 is currently operating on the text-002 models, and the 3rd version of GPT-3 is generally the one considered current. These benchmarks are for the original GPT3 model.

Re: DeepMind’s New Language Model, Chinchilla

#45

Post by gbasin »

isaacfrond wrote: post #26

They trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens while staying under a given compute budget. The results are modelled, and they pick the best one. Turns out the having a bit fewer tokens improves performance.

Thank you :)

Re: DeepMind’s New Language Model, Chinchilla

#46

Post by danuker »

eigenvalue wrote: post #25
ShamelessC wrote: post #5

Seems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin... > On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute. > Following the new scaling la…

According to the LessWrong post, the smaller model trained on more data performs better on most of the tasks, but it’s worse on “college level math” questions. I wonder why that is. Is it because the extra capacity of the larger model was used to basically memorize theorems? Or is it because the extra “brain power” let it model the math better? Oddly, one of the tasks that the smaller most outperformed the larger mod…

It might just be by chance: the initial weights of one model could have been lucky in some areas, and unlucky in others. There's no way to tell other than training again, which is a costly proposition.

Re: DeepMind’s New Language Model, Chinchilla

#47

Post by wiz21c »

I understand I can query such a model, one query at a time. But are there way to query these models with several queries in a row such that the N+1-th query benefit from the knowledge that was used to answer the N first questions ? Basically, following a conversation. For example, youtube subtitles can badly translate some terms but if "it" had in mind the overall subject of the video, then it'd probably pick the correct word...

Re: DeepMind’s New Language Model, Chinchilla

#48

Post by sillysaurusx »

sudosysgen wrote: post #41
VikingCoder wrote: post #33

Earlier quoted context omitted.

70 billion parameters... Is each of those a 4-byte float? So, is that 280 billion bytes of just parameters?

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Welcome to the party! I joined ML because I realized I could help. You can too. I bet you’re both already thinking of clever ways to deal with massive models from an infrastructure standpoint. That’s just one of hundreds of interesting problems.

Re: DeepMind’s New Language Model, Chinchilla

#49

Post by teraflop »

adamsmith143 wrote: post #42
mrfusion wrote: post #20

Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

On the other hand, consider the difficulty of taking massive amounts of data from the modern web and filtering out the subset that was actually generated by humans, rather than previous generations of language models.

Re: DeepMind’s New Language Model, Chinchilla

#50

Post by colordrops »

changoplatanero wrote: post #35
sirk390 wrote: post #34

Is outperforming GPT-3 still a good reference? It seems there are many models outperforming GPT-3 in the superglue benchmark: https://super.gluebenchmark.com/leaderboard/ GPT-3 is in position #21, with 71.8% score. The best model is at 91.2%. Note the human baseline in #6 with 89.8%

Aren’t most of the models at the top not suitable for text generation? That’s what makes gpt different from Bert

What are the models at the top used for? Excuse my ignorance.
Post reply on HN