I'd love to take a language model, load it up, and train it on everything I write in online learning mode. Does one need some massive hardware to do online learning with these models instead of just running the distilled final models?
DeepMind’s New Language Model, Chinchilla
101–110 of 145 posts
Re: DeepMind’s New Language Model, Chinchilla
#102Earlier quoted context omitted.
Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.
If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…
> It's clear that these models have orders of magnitude too much data already.
I have a toy disproof for your claim that this is clear.
Imagine that you are training a ML system using oracle access to Mum. The ML training system can request 10 million representative samples of Mum output, and then we could judge if the ML system has adequately reproduced Mum.
Now also imagine that Mum frequently tells people that Mum knows a 23 letter secret and while mum won't tell people what is outright, she'll answer queries like if a guess is lexographically higher or lower. We could even imagine that the ML has seen Mum's side of some interactions with her doing that.
Would the ML know Mum's secret? No.
Would a child that could interact with Mum? Yes-- after at most ceil(log_alphabet(23)) queries at most, if the child is efficient.
Learning in an interactive context is not the same as learning from written material, so you can't be sure that the fact that children learn english from less text means that a non-interactive ML system could english from the same amount. Q.E.D.
Now, if someone figures out how to efficiently train these natural language models with reinforcement learning...
Re: DeepMind’s New Language Model, Chinchilla
#103Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?
It implies our models are wrong. Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words. To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other langu…
My understanding is that the binding constraint in training these models is the quantity of computation they consume. While a human makes do with drastically less input data, we also have drastically more computational resources in our heads to work on the problem than Google is using to train its models.
Re: DeepMind’s New Language Model, Chinchilla
#104Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…
To add to this - they do this regularly, multiple times per week. While they do link to and acknowledge the source work, they do not make clear their writing is quoted or nearly quoted.
Re: DeepMind’s New Language Model, Chinchilla
#105Earlier quoted context omitted.
If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…
The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for example was trained on the Common Crawl dataset which would contain both Twitter and Reddit. >It's clear that these models have orders of magnitude too much data already. Seems like a strange claim. The scaling laws are showing that you can still make gains with m…
> Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters.
But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3.
GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective.
If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
Re: DeepMind’s New Language Model, Chinchilla
#106Earlier quoted context omitted.
Yeah, this implies backpropagation is deeply suboptimal.
That is certainly a possibility. The other (non-mutually exclusive) implications may also be that human language acquisition benefits from being part of a multi-task model. Or that the problem has been overreduced ie: human language acquisition cannot simply be distilled into a words-in->words-out problem and that vision/hearing are actually integral parts of language acquisition that cannot be left out. Or that mode…
> and that vision/hearing are actually integral parts of language acquisition
Deaf-blind authors would beg to differ.
But yes, a human brain is exposed to lots of other sensory input, and we know from other research that multi-modal models can learn shared representations that benefit from the knowledge of each domain.
In Transformer's favor, at least, they are far closer to tabula rasa than the human brain is and likely have to dedicate a lot of their training time to things that are otherwise "baked" into human brains. For example, humans come pre-packaged with V1 and V2 as part of their visual system, but CNNs and ViTs have to learn those filter packages from scratch.
I agree with you though. Human brains are able to take single instances of experiences and build a wealth of understanding from them in ways that even modern Transformer architectures are not yet able.
Re: DeepMind’s New Language Model, Chinchilla
#107Earlier quoted context omitted.
The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for example was trained on the Common Crawl dataset which would contain both Twitter and Reddit. >It's clear that these models have orders of magnitude too much data already. Seems like a strange claim. The scaling laws are showing that you can still make gains with m…
> Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters. But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3. GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of…
>But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3.
I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here.
>GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective.
This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so.
>If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.
Re: DeepMind’s New Language Model, Chinchilla
#108Re: DeepMind’s New Language Model, Chinchilla
#109Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…
We better get used to it. Because news companies will say an AI wrote it. No law allows suing an AI for plagiarism. Go prove something is not an AI.
Re: DeepMind’s New Language Model, Chinchilla
#110[deleted]