Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s New Language Model, Chinchilla

marktechpost.com

Post reply on HN
Print view View original

61–70 of 145 posts

Re: DeepMind’s New Language Model, Chinchilla

#61

Post by rolisz »

wiz21c wrote: post #47

I understand I can query such a model, one query at a time. But are there way to query these models with several queries in a row such that the N+1-th query benefit from the knowledge that was used to answer the N first questions ? Basically, following a conversation. For example, youtube subtitles can badly translate some terms but if "it" had in mind the overall subject of the video, then it'd probably pick the cor…

Yes. That's how you use GPT3: for the 2nd token, you feed in your prompt and the first token it returned. Then you feed it your prompt and the first two output tokens, and so on.

Re: DeepMind’s New Language Model, Chinchilla

#62

Post by VirusNewbie »

sillysaurusx wrote: post #55
ShamelessC wrote: post #40

Earlier quoted context omitted.

Gwern responded to a similar question in the comments section. (parent) > the fact that data and compute need to scale proportionally seems… like a big point in favor of NNs as memorizers/interpolators. (gwern) > Surely it's the opposite? The more bang you get out of each parameter, the less it looks like 'just' (whatever that means) memorization/interpolation. When you needed to increase parameters a lot, disproport…

This isn’t addressing their question. And Gwern’s goal here is to (incorrectly) try to get rid of the idea that models are just memorizing and interpolating, when in fact memorization and interpolation is what we all do, including models. He’s just bothered by the idea that people think of models as less than magic. On the other hand, https://twitter.com/model_mechanic/status/151297688118364569... is admittedly prett…

Why do you say they just memorize and interpret? I can teach GPT-2 new things, including new objects and their physical properties and it does a good job with that. That also means it has definitely not just regurgitated a matching sentence back to me.

Re: DeepMind’s New Language Model, Chinchilla

#63

Post by brrrrrm »

Delitio wrote: post #52
sudosysgen wrote: post #41

Earlier quoted context omitted.

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, h…

A good visual introduction to neural networks can be found here: https://playground.tensorflow.org

A parameter is a "weight" in this case (the lines drawn from neuron to neuron). The neurons are effectively runtime values or "activations." Parameters (weights) are updated during training and then set as constant during "inference" (also called "prediction").

There's unfortunately a ton of jargon and different groups use different words almost exclusively.

Re: DeepMind’s New Language Model, Chinchilla

#64

Post by gyang »

ShamelessC wrote: post #5

Seems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin... > On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute. > Following the new scaling la…

I think there remains an immense amount of such suboptimality still hanging from the tree, so to speak.

For example, our recent paper "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"[1] shows that even learning rate and initialization used by existing models are deeply wrong. By just picking them correctly (which involves some really beautiful mathematics), we can effectively double the model size of the GPT-3 6.7B model (to be comparable in quality to the 13B model across the suite of benchmark tasks).

Large neural networks behave in a way we are only beginning to understand well just because each empirical probe of any such model is so much more expensive and time consuming than typical models. But principled theory here can have a lot of leverage by pointing out the right direction to look, as it did in our work.

[1] http://arxiv.org/abs/2203.03466

Re: DeepMind’s New Language Model, Chinchilla

#65

Post by ritwikgupta »

Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site.

This seems highly unethical, and I'm surprised how they continue to operate.

Re: DeepMind’s New Language Model, Chinchilla

#66

Post by p1esk »

gyang wrote: post #64
ShamelessC wrote: post #5

Seems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin... > On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute. > Following the new scaling la…

I think there remains an immense amount of such suboptimality still hanging from the tree, so to speak. For example, our recent paper "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"[1] shows that even learning rate and initialization used by existing models are deeply wrong . By just picking them correctly (which involves some really beautiful mathematics), we can effectively d…

What do you think about the concept of "critical batch size"? https://openai.com/blog/science-of-ai/

Re: DeepMind’s New Language Model, Chinchilla

#67

Post by brrrrrm »

cshimmin wrote: post #32
sirk390 wrote: post #23

Earlier quoted context omitted.

And if every parameter is one byte, the minimum, it will take at least 70gb to save or share this model. So it's still way to big to package directly in a app.

From the paper, they are using bfloat16, so I guess two bytes. But distributing and "packaging into an app" are not at all of practical interest for these kinds of models. You (a consumer) would interact via some API service, with the model running on a hardware-accelerated compute cloud. In any case, during training (where the model is run in possibly large batches), and even during inference, the size of the parame…

> even during inference, the size of the parameters is completely dwarfed by the intermediate tensor representations

What makes you say this?

Re: DeepMind’s New Language Model, Chinchilla

#68

Post by mijoharas »

zarzavat wrote: post #60
adamsmith143 wrote: post #42

Earlier quoted context omitted.

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…

> It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.

I agree with your main point, but think this analogy isn't an apt one. If you want to see what particles are created at higher energies you kinda need the bigger particle accelerators. (This isn't to say that we shouldn't be investigating lower energy collisions, but at a certain point you do need "bigger colliders" to see new things)

Re: DeepMind’s New Language Model, Chinchilla

#69

Post by brrrrrm »

ritwikgupta wrote: post #65

Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…

Thanks for the heads up! In that case, I'd prefer not to share this link with peers. Do you have an alternative source with similar high-level content to share?

Re: DeepMind’s New Language Model, Chinchilla

#70

Post by georgehill »

ritwikgupta wrote: post #65

Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…

OP here - Thanks for sharing. I wasn't aware of this but despite this behavior, they are getting 600k visits.

https://www.similarweb.com/website/marktechpost.com/#overvie...

Post reply on HN