Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s New Language Model, Chinchilla

marktechpost.com

Post reply on HN
Print view View original

81–90 of 145 posts

Re: DeepMind’s New Language Model, Chinchilla

#81

Post by yldedly »

rictic wrote: post #74
sillysaurusx wrote: post #55

Earlier quoted context omitted.

This isn’t addressing their question. And Gwern’s goal here is to (incorrectly) try to get rid of the idea that models are just memorizing and interpolating, when in fact memorization and interpolation is what we all do, including models. He’s just bothered by the idea that people think of models as less than magic. On the other hand, https://twitter.com/model_mechanic/status/151297688118364569... is admittedly prett…

To rebut someone's argument you must address the argument and not just talk about them and their motivations From your comment a reader will understand that you think they're just memorizing and interpolating and that you disagree with gwern on this point, but you've given your reader nothing that argues in favor of your position Why should someone believe that models are just memorizing and interpolating?

It's impossible for a piecewise linear function to be anything other than linear outside the training sample. They are by their definition unable to do anything but interpolate.

Re: DeepMind’s New Language Model, Chinchilla

#82

Post by parhamn »

ritwikgupta wrote: post #65

Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…

I don’t know about this site, and I agree its unethical. But it does make me realize that I much prefer using language of the paper directly as opposed to having a non-expert poorly translate what your paper said. Especially given how papers put a lot of time in the accuracy and specificity of their language and word choices.

Would it also annoy you if they screwed up the interpretation of what you wrote? Is the alternative less reach of your work? For hard core research the tradeoffs are tougher it seems. If it is just a matter of non-nevermind, thats strictly messed up.

Re: DeepMind’s New Language Model, Chinchilla

#83

Post by lostmsu »

Delitio wrote: post #76
matt123456789 wrote: post #56

Earlier quoted context omitted.

A parameter is a scalar value, most of which are in the attention matrices and feedforward matrices, you also hear these called “weights”. Any intro to DL course will cover these in detail. I recommend started with Andrew Ng’s Coursera class on Intro to Machine Learning, although there may be better ones out there now.

Input parameter vs. weights then? I see tx

These networks (text models) usually have around a few thousand inputs.

Re: DeepMind’s New Language Model, Chinchilla

#84

Post by lostmsu »

sirk390 wrote: post #23
guipsp wrote: post #16

Earlier quoted context omitted.

You can think of a parameter as a number you can tweak while training. This network has 70B such numbers.

And if every parameter is one byte, the minimum, it will take at least 70gb to save or share this model. So it's still way to big to package directly in a app.

Have you seen modern games?

Re: DeepMind’s New Language Model, Chinchilla

#85

Post by lostmsu »

zarzavat wrote: post #60
adamsmith143 wrote: post #42

Earlier quoted context omitted.

Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.

If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…

I disagree with this take because you grok English not only from the text you read, but also from the context of physical world around you. And that context is enormous: assuming 8000x8000x2 vision with 3 color 1 byte channels at 24fps without compression, you get 3e+17 bytes (300 petabytes) of data along with your reading per year.

Re: DeepMind’s New Language Model, Chinchilla

#86

Post by kelseyfrog »

mrfusion wrote: post #20

Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?

It implies our models are wrong.

Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words.

To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other languages) 181 billion English words - or about 100 times more words than a human will hear by the time they reach adulthood. How is the human brain able to achieve a higher level of success with 1% of the data?

1. https://github.com/openai/gpt-3/blob/master/dataset_statisti...

Re: DeepMind’s New Language Model, Chinchilla

#87

Post by native_samples »

sillysaurusx wrote: post #48
sudosysgen wrote: post #41

Earlier quoted context omitted.

I'm fairly confident each of those is a 2-byte float, but yes that's over 100 GB of parameters.

Welcome to the party! I joined ML because I realized I could help. You can too. I bet you’re both already thinking of clever ways to deal with massive models from an infrastructure standpoint. That’s just one of hundreds of interesting problems.

Is 100GB of parameters really that large? 128GB of RAM on a server class machine is not unusual. Seems such a model could fit entirely in RAM.

Re: DeepMind’s New Language Model, Chinchilla

#88

Post by andbberger »

native_samples wrote: post #87
sillysaurusx wrote: post #48

Earlier quoted context omitted.

Welcome to the party! I joined ML because I realized I could help. You can too. I bet you’re both already thinking of clever ways to deal with massive models from an infrastructure standpoint. That’s just one of hundreds of interesting problems.

Is 100GB of parameters really that large? 128GB of RAM on a server class machine is not unusual. Seems such a model could fit entirely in RAM.

GPU memory is generally much smaller and more expensive

Re: DeepMind’s New Language Model, Chinchilla

#89

Post by native_samples »

priansh wrote: post #80
f38zf5vdt wrote: post #8

Earlier quoted context omitted.

There is probably a space-time trade off that needs to be explored in this space. It might be possible to preload the some of the most likely tokens to be selected next into the cache and/or RAM. These are glorified auto-complete algorithms that are poorly understood, as DeepMind's optimizations appear to show. For the English language, it is probable that there are only so many possible grammatically correct selecti…

I’ve been saying this for years, language models are the ML equivalent of the billionaire space race, it’s just a bunch of orgs with unlimited funding spending millions of dollars on compute to get more parameters than their rivals. It could be decades before we start to see them scale down or make meaningful optimizations. This paper is a good start but I’d be willing to bet everyone will ignore it and continue brea…

Is that fair? My Pixel phone seems to run nothing but ML models of various kinds and they run locally which is madness, pure madness. It can recognize songs and my speech without talking to the cloud at all. That's pretty much the definition of optimization!

Re: DeepMind’s New Language Model, Chinchilla

#90

Post by ralfd »

lostmsu wrote: post #85
zarzavat wrote: post #60

Earlier quoted context omitted.

If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…

I disagree with this take because you grok English not only from the text you read, but also from the context of physical world around you. And that context is enormous: assuming 8000x8000x2 vision with 3 color 1 byte channels at 24fps without compression, you get 3e+17 bytes (300 petabytes) of data along with your reading per year.

Blind children can learn english fine though. And there are areas highly unmaterial (mathematics) which people still reason about.
Post reply on HN