Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationality: From AI to Zombies

intelligence.org

Post reply on HN
Print view View original

61–70 of 142 posts

Re: Rationality: From AI to Zombies

#61

Post by maaku »

Strilanc wrote: post #52
xianshou wrote: post #5

This is a collection of Yudkowsky's blog posts on LessWrong, and it's making the rounds now because it forms the basis for most of Harry's rationalist insights in the extremely entertaining and recently finished Harry Potter and the Methods of Rationality. (The last chapter of HPMoR went up on Ultimate Pi Day - 3/14/15 - last week.) I picked it up on the advice of some friends, and while the editing is less than perf…

Given that this compilation was published on March 11th, and hpmor finished on March 14th, I don't think it's making the rounds now because of the timing of hpmor. Their finish/release dates are simply close together.

The release dates were purposefully close together.

Re: Rationality: From AI to Zombies

#62

Post by nsxwolf »

convexfunction wrote: post #43
Retric wrote: post #31

Earlier quoted context omitted.

People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?

Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real ; that was why it was a big deal.

You don't what? What policy? Information hazards? What?!

I feel like I just stumbled into a Scientology auditing session or something reading your comment.

Re: Rationality: From AI to Zombies

#63

Post by maxerickson »

ikeboy wrote: post #55
maxerickson wrote: post #29

Earlier quoted context omitted.

If you take seriously the idea of an infohazard, you have a problem, because the idea that information can be hazardous is itself hazardous information (at a minimum, it's a prompt for us sloppy thinkers to start imagining what sort of information might be hazardous). Something as innocuous as "Think safe thoughts!" enjoys a similar property.

And yet you posted this to the internet. Some people never learn, I guess.

Anything that somehow includes my saying "Think safe thoughts!" in a causal chain is already essentially inevitable. The future is not easily shaped.

Re: Rationality: From AI to Zombies

#64

Post by kbenson »

zyxley wrote: post #21
Houshalter wrote: post #17

Earlier quoted context omitted.

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilis…

Does changing some of the variables help?

I believe there is a secret government agency with the purpose of controlling the entire world will come about.

I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities.

In light of this, it's important to keep knowledge of this future event from people so they aren't targeted.

Whoops, I just posted this to a public forum.

Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.

Re: Rationality: From AI to Zombies

#65

Post by Houshalter »

pyrois wrote: post #36
philh wrote: post #22

Earlier quoted context omitted.

> EY's definition of good is ruthlessly utilitarian, Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.) With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?

I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing ete…

I think the dust speck argument would be better if first framed as a preference for yourself:

If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes.

I think most people would answer no. So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity (http://lesswrong.com/lw/hw/scope_insensitivity/), that our emotional reaction to things isn't diminished by even infinitesimally small quantities (see also Useless Medical Disclaimers http://lesswrong.com/lw/h4/useless_medical_disclaimers/).

I think I can make that position seem more reasonable if framed another way: Would you risk a 1/(3^^^3) chance at 50 years of torture, in exchange for getting rid of one dust speck out of your eye? I think most people would say yes. You couldn't even get out of bed in the morning if you weren't willing to take even incredibly small risks.

There is a much higher probability than 1/(3^^^3) that you could get in a car accident with injuries that cause 50 years of incredible pain, yet you will still probably risk driving for even trivial things.

And if you lived for 3^^^3 lifetimes and took this risk each time, you would likely suffer 50 years of torture during at least one lifetime.

Re: Rationality: From AI to Zombies

#66

Post by Houshalter »

spacehome wrote: post #50
Retric wrote: post #31

Earlier quoted context omitted.

People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?

The visible universe? Some back-of-the-napkin estimates are that no more than 2^500 bits will be flipped in the region of space we can see, but there's a crucial assumption about minimum temperature you need to make to make that that estimate finite. On the other hand, there's some pretty compelling reasoning that the entire universe is actually infinite in extent, so ... infinite computational power?

What an absurd thing to consider. There is no possibility of building a computer the size of the universe. This website is full of crazy people talking about crazy things!

Re: Rationality: From AI to Zombies

#67

Post by maxerickson »

TeMPOraL wrote: post #53
maxerickson wrote: post #40

Earlier quoted context omitted.

Yet in this same thread here there is a discussion of torture and dust specks. As if the mystical power to prevent dust specks is close enough to reality to be interesting. I wonder how many people that fed that question into their mentation have considered sabotaging a coal plant. Obviously, that's ethically more complicated than the thought experiment, but it follows the outline. (If you believe that the substitute…

It's actually a fair question. But 1) I don't believe substitutes for coal power are immediately available, and taking out a plant would likely lead to many deaths (think: accidents caused by sudden blackout, rural hospitals without enough fuel for backup power generators) and inconvenience (think: areas with interrupted electrical service for months as people try to fix the grid and bring in additional power sources…

My point is largely that I don't think you learn much until you actually go ahead and twist the knife yourself (or not!).

Short of that, it's just a mind puzzle, "how does what I think my beliefs are match up with what I think my beliefs are" or so.

Re: Rationality: From AI to Zombies

#68

Post by pyrois »

Houshalter wrote: post #65
pyrois wrote: post #36

Earlier quoted context omitted.

I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing ete…

I think the dust speck argument would be better if first framed as a preference for yourself: If you were going to live for 3^^^3 lifetimes, would you like to be tortured for 50 years in exchange for one fewer dust specks in your eye during each of those lifetimes. I think most people would answer no . So the debate has nothing to do with morality in general. Instead the debate is about Scope Insensitivity ( http://l…

Yes, I agree, I would take the risk, but those are two entirely different things. It's not the same argument framed in a different way at all.

In Yudkowsky's argument, it's the option between one person _absolutely guaranteed_ to have 50 years of torture, vs 3^^^3 people _absolutely guaranteed_ to be dust-moted. I think the guarantee changes things significantly.

Re: Rationality: From AI to Zombies

#69

Post by TeMPOraL »

maxerickson wrote: post #67
TeMPOraL wrote: post #53

Earlier quoted context omitted.

It's actually a fair question. But 1) I don't believe substitutes for coal power are immediately available, and taking out a plant would likely lead to many deaths (think: accidents caused by sudden blackout, rural hospitals without enough fuel for backup power generators) and inconvenience (think: areas with interrupted electrical service for months as people try to fix the grid and bring in additional power sources…

My point is largely that I don't think you learn much until you actually go ahead and twist the knife yourself (or not!). Short of that, it's just a mind puzzle, "how does what I think my beliefs are match up with what I think my beliefs are" or so.

I always found it as the "how much my gut reaction matches with what the mathematical model suggests" kind of puzzle - useful for both exploring your own biases and the limitations of the model.

Re: Rationality: From AI to Zombies

#70

Post by maaku »

emiliobumachar wrote: post #6

I am currently reading, and loving, Yudkowsky's introduction to Quantum Mechanics from the blog: http://lesswrong.com/lw/pc/quantum_explanations/ Does anyone know whether that sequence has made it to one of these books?

There is a much reduced and heavily edited sequence on quantum mechanics in the ebook. I have not yet read the ebook version and can't tell you how they differ, but the editor did state upfront that it was one of the sections which changed the most.

Regarding the LessWrong/OvercomingBias version, I must say that you should take the blog posts with a huge, gigantic, over-sized helping of salt. Speaking as someone who majored in physics because I love the field even though I knew it wouldn't help my comp sci career, Yudkowsky does not have the necessary background to speak with authority on this topic and it shows. He makes a few outright mistakes and a lot more overly broad statements, he ignores many validly competing alternative explanations, and as a result the grand philosophical conclusions reached are left unjustified (and as an editorial aside, probably bunk IMHO).

The important result of analyzing quantum mechanics as a case study in rationality is the valid point that if like most people (and sadly, most physics professors) you think quantum mechanics is weird, that should be very concerning. Quantum mechanics is real: it very accurately describes reality. If you have a problem with quantum mechanics, you have a problem with reality, and that's about as anti-rationalist as you can get. To improve, find out what part of you rejects the reality of quantum mechanics in spite of experimental evidence, and censor that.

Once you've achieved that step, you can skip the rest IMHO.

Post reply on HN