91–100 of 145 posts
Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?
It implies our models are wrong. Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words. To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other langu…
Earlier quoted context omitted.
Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.
If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…
>It's clear that these models have orders of magnitude too much data already.
Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters.
>It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.
This is literally true though, couldn't find the Higgs without the LHC and most GUT candidates would only start being ruled out at high energy levels.
Earlier quoted context omitted.
Probably not an issue just yet, think of how much data is generated by Twitter on a daily basis for example.
On the other hand, consider the difficulty of taking massive amounts of data from the modern web and filtering out the subset that was actually generated by humans, rather than previous generations of language models.
Earlier quoted context omitted.
It implies our models are wrong. Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words. To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other langu…
Yeah, this implies backpropagation is deeply suboptimal.
If these things get put on specialized hardware for inference with much lower energy costs, the world will never be the same.
Imagine any diffusion-style text-to-image model on specialized ASIC hardware.
If you mean putting the model weights into gates directly, it’d be useless because users would get bored of the model as soon as they figured out what its style looked like. Also, large models can memorize their training data so eventually you’ll get it to output something copyrighted.
Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?