Rationality: From AI to Zombies • Hacker News Board

Re: Rationality: From AI to Zombies

#21

Post by zyxley » Fri, Mar 20, 2015, 3:31 PM UTC

Houshalter wrote: post #17

danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe.

I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?

Re: Rationality: From AI to Zombies

#22

Post by philh » Fri, Mar 20, 2015, 3:39 PM UTC

danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

> EY's definition of good is ruthlessly utilitarian,

Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.)

With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?

Re: Rationality: From AI to Zombies

#23

Post by unbudgingprawn » Fri, Mar 20, 2015, 3:42 PM UTC

zyxley wrote: post #21

Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilis…

It seems like he deeply believes in it. I don't think he realizes it's just a modern version of Pascal's wager

Re: Rationality: From AI to Zombies

#24

Post by TeMPOraL » Fri, Mar 20, 2015, 3:42 PM UTC

zyxley wrote: post #21

Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilis…

> caused me to yell quite loudly at Roko for violating ethics given his own premises

As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.

Re: Rationality: From AI to Zombies

#25

Post by TeMPOraL » Fri, Mar 20, 2015, 3:44 PM UTC

zyxley wrote: post #19

danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).

Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".

Re: Rationality: From AI to Zombies

#26

Post by mori » Fri, Mar 20, 2015, 3:45 PM UTC

zyxley wrote: post #21

Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilis…

Well, yes? If Roko's basilisk was true, it'd be unethical to spread. It turned out to not be true, but if Roko was reasoning from the premise of it being true, or even possibly-true, then the action was still unethical.

Re: Rationality: From AI to Zombies

#27

Post by philh » Fri, Mar 20, 2015, 3:46 PM UTC

zyxley wrote: post #21

Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilis…

More like, "he believed that he'd accidentally invented an eternal torture machine and then he tried to turn it on, and I yelled at him even though the eternal torture machine didn't work."

Re: Rationality: From AI to Zombies

#28

Post by mori » Fri, Mar 20, 2015, 3:48 PM UTC

zyxley wrote: post #19

danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).

What, preference utilitarianism? Afaik, that does get discussed, especially in the Fun Theory sequence iirc. If you want some more discussion on torture vs. dust specks, I think nostalgebraist on tumblr had a discussion of it with slatestarscratchpad (again, iirc).

Re: Rationality: From AI to Zombies

#29

Post by maxerickson » Fri, Mar 20, 2015, 3:48 PM UTC

danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

If you take seriously the idea of an infohazard, you have a problem, because the idea that information can be hazardous is itself hazardous information (at a minimum, it's a prompt for us sloppy thinkers to start imagining what sort of information might be hazardous).

Something as innocuous as "Think safe thoughts!" enjoys a similar property.

Re: Rationality: From AI to Zombies

#30

Post by philh » Fri, Mar 20, 2015, 3:50 PM UTC

TeMPOraL wrote: post #25

zyxley wrote: post #19

Earlier quoted context omitted.

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).

Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".

I think he does, although he doesn't say so explicitly in the original post. http://lesswrong.com/lw/kn/torture_vs_dust_specks/

At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.