Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationality: From AI to Zombies

intelligence.org

Post reply on HN
Print view View original

51–60 of 142 posts

Re: Rationality: From AI to Zombies

#51

Post by pyrois »

Vraxx wrote: post #48
pyrois wrote: post #36

Earlier quoted context omitted.

I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing ete…

That last point about the models actually captures a rather interesting point. Generally I agree that if the model isn't giving results that match observation than it should be concluded that the model is incorrect rather than the observation was flawed. This has worked very well as the scientific method, but I think that it is hinged on the fact that these observations are objective and verifiable. Now I happen to a…

Fair enough. I should have said "the model fails to accurately reflect my sense of morality".

Re: Rationality: From AI to Zombies

#52

Post by Strilanc »

xianshou wrote: post #5

This is a collection of Yudkowsky's blog posts on LessWrong, and it's making the rounds now because it forms the basis for most of Harry's rationalist insights in the extremely entertaining and recently finished Harry Potter and the Methods of Rationality. (The last chapter of HPMoR went up on Ultimate Pi Day - 3/14/15 - last week.) I picked it up on the advice of some friends, and while the editing is less than perf…

Given that this compilation was published on March 11th, and hpmor finished on March 14th, I don't think it's making the rounds now because of the timing of hpmor. Their finish/release dates are simply close together.

Re: Rationality: From AI to Zombies

#53

Post by TeMPOraL »

maxerickson wrote: post #40
TeMPOraL wrote: post #18

Earlier quoted context omitted.

The lesson I learned from Roko's Basilisk event that I think Eliezer and most of the LW community also endorses is that if you're not careful, applying some models too far can have those "extreme, nay dire consequences for certain thought experiments" - so it's best not to commit too far to those methods, because they may have some caveats, or you might just have applied them wrong.

Yet in this same thread here there is a discussion of torture and dust specks. As if the mystical power to prevent dust specks is close enough to reality to be interesting. I wonder how many people that fed that question into their mentation have considered sabotaging a coal plant. Obviously, that's ethically more complicated than the thought experiment, but it follows the outline. (If you believe that the substitute…

It's actually a fair question. But 1) I don't believe substitutes for coal power are immediately available, and taking out a plant would likely lead to many deaths (think: accidents caused by sudden blackout, rural hospitals without enough fuel for backup power generators) and inconvenience (think: areas with interrupted electrical service for months as people try to fix the grid and bring in additional power sources); 2) downsides of a coal plant aren't that bad in comparison.

Note that dustspeck vs. torture example involved 3^^^3 dustspecks, which is more than number of atoms in the universe raised to the power of number of atoms in the universe. It was purposefully chosen as such an extreme number, as to clearly outweight the utilitarian calculation in favour of torture - so that the example could be used to study our moral intuitions.

Re: Rationality: From AI to Zombies

#54

Post by TeMPOraL »

Strilanc wrote: post #52
xianshou wrote: post #5

This is a collection of Yudkowsky's blog posts on LessWrong, and it's making the rounds now because it forms the basis for most of Harry's rationalist insights in the extremely entertaining and recently finished Harry Potter and the Methods of Rationality. (The last chapter of HPMoR went up on Ultimate Pi Day - 3/14/15 - last week.) I picked it up on the advice of some friends, and while the editing is less than perf…

Given that this compilation was published on March 11th, and hpmor finished on March 14th, I don't think it's making the rounds now because of the timing of hpmor. Their finish/release dates are simply close together.

I think it still may be - people are posting this around as an additional reading for those who liked HPMOR; number of people who read that fanfic is most likely much greater than the number of people that read the Sequences.

Re: Rationality: From AI to Zombies

#55

Post by ikeboy »

maxerickson wrote: post #29
danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

If you take seriously the idea of an infohazard, you have a problem, because the idea that information can be hazardous is itself hazardous information (at a minimum, it's a prompt for us sloppy thinkers to start imagining what sort of information might be hazardous). Something as innocuous as "Think safe thoughts!" enjoys a similar property.

And yet you posted this to the internet. Some people never learn, I guess.

Re: Rationality: From AI to Zombies

#56

Post by TeMPOraL »

zyxley wrote: post #41
philh wrote: post #37

Earlier quoted context omitted.

Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture. Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?

> Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.

One reason is that you'd need to prove that the lastic impact is in fact zero. 3^^^3 dust specks is enough for one of them to rub something in a wrong way that rubs something else in the wrong way that gives a person cancer.

Re: Rationality: From AI to Zombies

#57

Post by bryondowd »

pyrois wrote: post #36
philh wrote: post #22

Earlier quoted context omitted.

> EY's definition of good is ruthlessly utilitarian, Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.) With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?

I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing ete…

There aren't 3^^^3 atoms in the universe, let alone people. I think it's absurd to complain about a system not holding up when stretched far beyond the boundaries of the universe.

If you decrease the number to even the number of humans that ever existed/exist/will exist, then it returns to being negligible in relation to torture.

The problem with an if -> then, is when the 'if' is flawed, it doesn't really matter what the 'then' is. The human mind can't grasp 3^^^3 dust motes because it doesn't exist in any meaningful way.

Re: Rationality: From AI to Zombies

#58

Post by Strilanc »

zyxley wrote: post #19
danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).

It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong.

Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture.

> A system that is optimizing a function of n variables, where the objective depends on a subset of size k http://edge.org/conversation/jaron_lanier-the-myth-of-ai

Re: Rationality: From AI to Zombies

#59

Post by mdda »

philh wrote: post #45
DanAndersen wrote: post #38

Earlier quoted context omitted.

Right, that's why I reject the assumption that you can multiply it like that. Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? The only reasonable outcome in whic…

> What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects. But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, th…

(I'm not totally clear whether it was meant to be 1penny for each speck avoided or 1penny vs all specks.). But two follow-up questions :

a) would you press the 7bn specks button for an individual 'reward' of $70million?

b) would you press the N-specks button if given a choice between that and the 1-torture button?

Re: Rationality: From AI to Zombies

#60

Post by ikeboy »

pyrois wrote: post #36
philh wrote: post #22

Earlier quoted context omitted.

> EY's definition of good is ruthlessly utilitarian, Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.) With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?

I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing ete…

What do you say to the argument in http://lesswrong.com/lw/n3/circular_altruism/:

>But let me ask you this. Suppose you had to choose between one person being tortured for 50 years, and a googol people being tortured for 49 years, 364 days, 23 hours, 59 minutes and 59 seconds. You would choose one person being tortured for 50 years, I do presume; otherwise I give up on you.

>And similarly, if you had to choose between a googol people tortured for 49.9999999 years, and a googol-squared people being tortured for 49.9999998 years, you would pick the former.

>A googolplex is ten to the googolth power. That's a googol/100 factors of a googol. So we can keep doing this, gradually - very gradually - diminishing the degree of discomfort, and multiplying by a factor of a googol each time, until we choose between a googolplex people getting a dust speck in their eye, and a googolplex/googol people getting two dust specks in their eye.

>If you find your preferences are circular here, that makes rather a mockery of moral grandstanding. If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, you may have fun driving, but you aren't going anywhere. Maybe you think it a great display of virtue to choose for a googolplex people to get dust specks rather than one person being tortured. But if you would also trade a googolplex people getting one dust speck for a googolplex/googol people getting two dust specks et cetera, you sure aren't helping anyone. Circular preferences may work for feeling noble, but not for feeding the hungry or healing the sick.

Post reply on HN