Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationality: From AI to Zombies

intelligence.org

Post reply on HN
Print view View original

41–50 of 142 posts

Re: Rationality: From AI to Zombies

#41

Post by zyxley »

philh wrote: post #37
DanAndersen wrote: post #34

Earlier quoted context omitted.

Would you be able to help me understand your reasoning? From my perspective, "torture vs dust specks" doesn't seem to be a thing you can just "shut up and multiply" on, because the impact of torture vs the impact of a dust speck seem fundamentally different. To me the negative impact of pain is not just the sensory experience on a purely linear scale, but the effect it has on a person's life even afterward. It seems…

Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture. Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?

> Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero

Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.

Re: Rationality: From AI to Zombies

#43

Post by convexfunction »

Retric wrote: post #31
Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?

Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real; that was why it was a big deal.

Re: Rationality: From AI to Zombies

#44

Post by MBlume »

zyxley wrote: post #21
Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilis…

If you think you've discovered a serious info-hazard and your first action is to post it publicly, you are not someone I want frequenting the same online spaces I frequent. Whether that particular info-hazard is, upon examination, remotely hazardous is aside from the point.

(If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)

Re: Rationality: From AI to Zombies

#45

Post by philh »

DanAndersen wrote: post #38
philh wrote: post #37

Earlier quoted context omitted.

Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture. Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?

Right, that's why I reject the assumption that you can multiply it like that. Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? The only reasonable outcome in whic…

> What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering?

I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects.

But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny.

Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die.

Which do you choose? Sixty years from now, which do you wish you'd chosen?

> I wouldn't pay a penny to avoid the dust speck

Do you mean just your own dust speck? The question was all 3^^^3 of them.

If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)

Re: Rationality: From AI to Zombies

#46

Post by Symmetry »

zyxley wrote: post #19
danbmil99 wrote: post #14

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of "good" in order to weigh and compare outcomes. EY's definition of good is ruthlessly utilitarian, and that has some extreme, nay dire consequences for certain thought experiments. Consume this philosophy at your own risk.

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).

3^^^3 people isn't "a very large number." The population of Earth or a galaxy full of inhabited planets would be a very large number. 3^^^3 is an incomprehensibly large number which is why it can lead to odd outcomes. If it were merely something like everyone on Earth suffering a dust speck then of course you choose dust.

Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.

Re: Rationality: From AI to Zombies

#47

Post by strnfcpy »

zyxley wrote: post #32
TeMPOraL wrote: post #25

Earlier quoted context omitted.

Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".

He does. He says so explicitly in the comments. > I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.

If you opt to remove the SPECKS, you would have to change some law of nature or some property of matter, which will have other consequences for the amount of suffering in this world. As you cannot reliably predict or calculate those consequences, you cannot solve this problem.

Re: Rationality: From AI to Zombies

#48

Post by Vraxx »

pyrois wrote: post #36
philh wrote: post #22

Earlier quoted context omitted.

> EY's definition of good is ruthlessly utilitarian, Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.) With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?

I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing ete…

That last point about the models actually captures a rather interesting point. Generally I agree that if the model isn't giving results that match observation than it should be concluded that the model is incorrect rather than the observation was flawed. This has worked very well as the scientific method, but I think that it is hinged on the fact that these observations are objective and verifiable.

Now I happen to agree with you that I feel this dust result is incorrect, but morality is so undefined and subjective that I can't help but disagree with your conclusion that the model must be wrong. The entire moral question of the dust vs torture doesn't have one defined answer so we can't really conclude anything more than opinions about the model

Re: Rationality: From AI to Zombies

#49

Post by spacehome »

Strilanc wrote: post #35
philh wrote: post #22

Earlier quoted context omitted.

> EY's definition of good is ruthlessly utilitarian, Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.) With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?

(Different person here.) The most common example I see is the dust-specks vs torture thing[1], where Eliezer thinks increasing dust specks eventually outweighs the torture and various other people think torture outweighs any number of dust specks. Eliezer is basing his position on taking a few decision theory axioms as obviously true, and others are basing their position on taking their moral intuitions as obviously…

Yea, I agree with your interpretation on the root cause of the disagreement. I don't mean to weigh in one way or the other, but I would like to point out that people's mathematical intuition when it comes to infinity or obscenely large numbers can be arbitrarily wrong. It takes a lot of training to even recognize the mistakes you're making, let alone correct for them.

Re: Rationality: From AI to Zombies

#50

Post by spacehome »

Retric wrote: post #31
Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?

The visible universe? Some back-of-the-napkin estimates are that no more than 2^500 bits will be flipped in the region of space we can see, but there's a crucial assumption about minimum temperature you need to make to make that that estimate finite.

On the other hand, there's some pretty compelling reasoning that the entire universe is actually infinite in extent, so ... infinite computational power?

Post reply on HN