Are language models deprived of electric sleep?
1–10 of 95 posts
Re: Are language models deprived of electric sleep?
#2I’m well aware that this is an unpopular opinion.
When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet.
People read too much into this.
The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sentence-was-on-the-internet).
Re: Are language models deprived of electric sleep?
#3I’m well aware that this is an unpopular opinion. When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet. People read too much into this. The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sen…
The whole show in ML these days is in the infra around massively distributed, redundant, tunable, train the same fucking transformer we’ve all been using since 2017.
Re: Are language models deprived of electric sleep?
#4I’m well aware that this is an unpopular opinion. When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet. People read too much into this. The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sen…
The whole show in ML these days is in the infra around massively distributed, redundant, tunable, train the same fucking transformer we’ve all been using since 2017.
What about diffusion models? (And all the other things with less publicity that aren’t that)
Re: Are language models deprived of electric sleep?
#5Earlier quoted context omitted.
The whole show in ML these days is in the infra around massively distributed, redundant, tunable, train the same fucking transformer we’ve all been using since 2017.
What about diffusion models? (And all the other things with less publicity that aren’t that)
Sure, I’m obviously being a big glib.
So it’s transformers plus let’s force some loss into the super-resolution pipeline.
“Residual” sounds a lot fancier than X = fn + X, and “Diffusion” sounds a lot fancier than “let’s do deconv repeatedly and jam a loss term in there”.
My point is all of this stuff is on Huggingface. You/I/We just don’t have 5k A100s for 6 months, so we can’t play.
Re: Are language models deprived of electric sleep?
#6I’m well aware that this is an unpopular opinion. When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet. People read too much into this. The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sen…
I completely agree.
Nonetheless, this approach of taking a fairly complex writing prompt and simply regurgitating contextually similar text from the internet often yields 100% meaningful results. This suggests that our intelligence is a lot more mimetic than we'd like to believe, which is perhaps an even more unpopular opinion.
Re: Are language models deprived of electric sleep?
#7I’m well aware that this is an unpopular opinion. When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet. People read too much into this. The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sen…
I completely agree. Nonetheless, this approach of taking a fairly complex writing prompt and simply regurgitating contextually similar text from the internet often yields 100% meaningful results. This suggests that our intelligence is a lot more mimetic than we'd like to believe, which is perhaps an even more unpopular opinion.
I tend to agree that the math gets a lot simpler if the sun doesn’t rotate around a mystical axiom that humans are “special”.
Re: Are language models deprived of electric sleep?
#8Just a side note, I was showing some friends gpt3 and asking some trivial questions, trying to impress them, when one of them asked me: "Is it any different from like asking google for the same information?" I mean, In a pure naive and practical point of view she wasn't that wrong.
Re: Are language models deprived of electric sleep?
#9I’m well aware that this is an unpopular opinion. When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet. People read too much into this. The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sen…
The whole show in ML these days is in the infra around massively distributed, redundant, tunable, train the same fucking transformer we’ve all been using since 2017.
If you can find a more efficient or effective approach than this “redundant” one you’ll be rich. There’s a lot of smart people working in this space.
Re: Are language models deprived of electric sleep?
#10Earlier quoted context omitted.
I completely agree. Nonetheless, this approach of taking a fairly complex writing prompt and simply regurgitating contextually similar text from the internet often yields 100% meaningful results. This suggests that our intelligence is a lot more mimetic than we'd like to believe, which is perhaps an even more unpopular opinion.
I tend to agree that the math gets a lot simpler if the sun doesn’t rotate around a mystical axiom that humans are “special”.
High-energy physicists are about as serious as it gets, but they’re still dragging their feet around many worlds because it has uncomfortable implications for consciousness and free will.
Homo Sapiens has a serious agenda around being something other than the smartest chimpanzee.