111–120 of 145 posts
Earlier quoted context omitted.
That may be true, but in the US there are typically math courses before calculus.
But then we get a calculator.
Anyway it distracts from the point so it's not relevant.
Earlier quoted context omitted.
I disagree with this take because you grok English not only from the text you read, but also from the context of physical world around you. And that context is enormous: assuming 8000x8000x2 vision with 3 color 1 byte channels at 24fps without compression, you get 3e+17 bytes (300 petabytes) of data along with your reading per year.
Blind children can learn english fine though. And there are areas highly unmaterial (mathematics) which people still reason about.
Is outperforming GPT-3 still a good reference? It seems there are many models outperforming GPT-3 in the superglue benchmark: https://super.gluebenchmark.com/leaderboard/ GPT-3 is in position #21, with 71.8% score. The best model is at 91.2%. Note the human baseline in #6 with 89.8%
It is if you outperform it with much fewer parameters
Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…
We better get used to it. Because news companies will say an AI wrote it. No law allows suing an AI for plagiarism. Go prove something is not an AI.
Earlier quoted context omitted.
That is certainly a possibility. The other (non-mutually exclusive) implications may also be that human language acquisition benefits from being part of a multi-task model. Or that the problem has been overreduced ie: human language acquisition cannot simply be distilled into a words-in->words-out problem and that vision/hearing are actually integral parts of language acquisition that cannot be left out. Or that mode…
> and that vision/hearing are actually integral parts of language acquisition Deaf-blind authors would beg to differ. But yes, a human brain is exposed to lots of other sensory input, and we know from other research that multi-modal models can learn shared representations that benefit from the knowledge of each domain. In Transformer's favor, at least, they are far closer to tabula rasa than the human brain is and li…
Does this imply we will run out of data to keep up with larger model sizes? Is there much more data out there than what they’re already using?
It implies our models are wrong. Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words. To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other langu…
The most obvious answer is "the human brain uses a shit-ton more compute", for 18+ years as well.
We spend data, which we have in abundance, to save on compute, which we do not. Even at the most generous low-end estimates of the human brain's computing power, we are only barely there; on the high-end estimates that people in love with the ineffable mysteries of the brain love to cite, we are multiple orders of magnitude away from even the biggest supercomputers matching the brain. So no matter which way you slice it, we are extremely compute-poor.
Feeding a lot of data through an extremely lightweight optimizer like first-order SGDs is one way to cope with lacking compute: https://www.gwern.net/docs/ai/scaling/2013-bottou.pdf Bottou asks why (even in 2013!) is SGD so hard to dethrone when we can empirically see plenty of optimizers like second-order gradient descent algorithms which can beat SGD quite solidly? His observation is that while they are much better than SGD in terms of iterations or _n_, they lose in compute/wallclock because SGD can just go-brrrr through the data much faster than they can.
Earlier quoted context omitted.
If you want to teach your kid to learn English, and they came back to you and said "Dad/mum, I finished reading the entire internet but I still don't understand English fully" , would you say "OK son, now go and stare at the Twitter firehouse until you grok perfect English" ? It's clear that these models have orders of magnitude too much data already. It somewhat reminds me of the proposals for larger and larger coll…
The general point is that there is a huge volume of training data generated daily not that Twitter is a great source of it. Though I believe that GPT-3 for example was trained on the Common Crawl dataset which would contain both Twitter and Reddit. >It's clear that these models have orders of magnitude too much data already. Seems like a strange claim. The scaling laws are showing that you can still make gains with m…
I have to ask, why call it that? I had a chuckle once I saw the name.