Are language models deprived of electric sleep? • Hacker News Board

Re: Are language models deprived of electric sleep?

#91

Post by benreesman » Tue, Jul 26, 2022, 7:31 AM UTC

NavinF wrote: post #82

benreesman wrote: post #19

Earlier quoted context omitted.

Haha well played. I’ll counter your non-sequitor with an appeal to authority ;) I used to do this for a living.

If you’re a language model, congrats on hitting 0.00 perplexity on the HN comments dataset. Your sponsors would be proud :P

If I was going to hook a big language model to HN I wouldn’t do it under my real name.

But… I take your meaning. I think I was subconsciously channeling OPT because that’s what the thread is about.

I agree it’s the kind of thing a fine-tuned HN chat bot would say all the friggin time.

Re: Are language models deprived of electric sleep?

#92

Post by benreesman » Tue, Jul 26, 2022, 7:38 AM UTC

telotortium wrote: post #78

benreesman wrote: post #2

I’m well aware that this is an unpopular opinion. When your game plan is take a multi-head attention transformer and overfit it on as much of the Internet as we have A100s to fit it on, you’re going to get a distribution, and if you sample from it, it’s going to sound like the Internet. People read too much into this. The technology is obviously powerful borderline dangerous, but it’s still just maximizing P(this-sen…

You better watch what you're saying - soon the champion (in the medieval sense) of ML, Gwern, will come to defend his liege's honor. I see the other champion, Veedrac, has already come out :)

Oh I took a licking from @gwern (who I do respect but isn’t as insightful as his fame would imply) a few weeks ago for denying that the Sigularity Is Near.

Being Mr Who Ever Heard of This Guy myself, I took it as a compliment. :)

@veedrac on the other hand, who is smart, pretty clearly got the worse on that friendly little skirmish.

Re: Are language models deprived of electric sleep?

#93

Post by benreesman » Tue, Jul 26, 2022, 7:53 AM UTC

legalcorrection wrote: post #69

benreesman wrote: post #57

Earlier quoted context omitted.

You strongly disagree that other intelligent, social, creative animals are built along similar lines to Homo sapiens? You really think that we’re a special case, that a difference in degree has become a difference in kind? I personally experience a feeling that I’m conscious subjectively, but I have no evidence that I’m any more or less motivated by pleasure or pain or community than a dolphin is. Where do we draw th…

I don't mean to suggest that animals don't also have consciousness, or that it's not important to explaining their behavior too.

Why not go the other way and just admit that Descartes gave us the Cartesian plane (among other things) but was at best a product of his time with: “I think therefore I am”.

Descartes was a genius, but he was no Alan Turing, and Alan-fucking-Turing got it wrong on the most famous thing named after him (among the lay population at least). The Turing Test was a great idea, but it’s now trivially useless.

Humans are special to (mostly) themselves and (substantially) other humans.

They are not special to the universe. We’ve had this argument, it was called the “Inquisition” at least once, and we eventually cleared up once and for all what celestial body rotates around the bigger one.

Re: Are language models deprived of electric sleep?

#94

Post by the8472 » Wed, Jul 27, 2022, 3:05 PM UTC

trention wrote: post #68

the8472 wrote: post #48

Earlier quoted context omitted.

Part of the evaluation procedures for the models like these is checking whether the output they produce is present in the training set. The fraction of samples being verbatim reurgitations is low unless you prompt-engineer your way to that, copilot being able to recite the fast inverse square root function being a common example. But the ability to recite some samples is not proof of inability to create novel outputs…

>The fraction of samples being verbatim reurgitations is low Copilot spitting out a function out of its training data with changed variable names to match those in your file(s) - and no one is actually testing what proportions of results those are - is still regurgitation.

> and no one is actually testing what proportions of results those are

The idea that models could possibly overfit the training data is hardly a new idea. It's standard practice to test for that. Check section 7 of the PaLM paper for example. https://arxiv.org/abs/2204.02311

Re: Are language models deprived of electric sleep?

#95

Post by legalcorrection » Wed, Aug 03, 2022, 9:40 PM UTC

benreesman wrote: post #93

legalcorrection wrote: post #69

Earlier quoted context omitted.

I don't mean to suggest that animals don't also have consciousness, or that it's not important to explaining their behavior too.

Why not go the other way and just admit that Descartes gave us the Cartesian plane (among other things) but was at best a product of his time with: “I think therefore I am”. Descartes was a genius, but he was no Alan Turing, and Alan-fucking-Turing got it wrong on the most famous thing named after him (among the lay population at least). The Turing Test was a great idea, but it’s now trivially useless. Humans are spe…

Your position sounds much more religious and dogmatic than those held by the people you are arguing against.