Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Most Terrifying Thought Experiment of All Time

slate.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: The Most Terrifying Thought Experiment of All Time

#51

Post by crdoconnor »

a_bonobo wrote: post #37
crdoconnor wrote: post #33

Earlier quoted context omitted.

* Because Pascal's wager could apply to any possible variation of supreme being - including non-sensical ones and ones that will punish you if you DO believe in them. * Roko's Basilisk isn't really a deity, it's a powerful AI with the capability of simulating you and everybody else (or possibly just you). * Pascal's wager does concern itself with the possibility that you are currently in a simulation.

>Roko's Basilisk isn't really a deity, it's a powerful AI with the capability of simulating you and everybody else (or possibly just you). At the risk of sounding flippant: What's the difference? If the AI is so powerful that it can simulate my thoughts (which would imply that thoughts and all of life is 100% deterministic), it has to be so powerful that it's practically omnipotent, and therefore, a God.

From what I understood, it doesn't require 100% determinism OR a perfect simulation for the thought experiment to hold.

It just needs enough to sort most of the humans into "will help basilisk come into existence" and "won't help basilisk". It probably doesn't even need to be 100% accurate at that.

I don't think being able to imperfectly simulate thoughts requires Godlike ability, either. I could see it happening in my life time.

Assuming I'm not being simulated right now, that is :/

Re: The Most Terrifying Thought Experiment of All Time

#52

Post by jcbrand »

teekert wrote: post #11

"it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes." This makes him Yudkowsky the perfect lab animal! Think of the things we can learn, the lives we can save by performing experiments (that most think are unethical) on him! As for the box choice? Just start chopping through the alien's head, the unknown that l…

"it is preferable to torture a single person for 50 years than for a sufficient number of people (to be fair, a lot of people) to get dust specks in their eyes."

Only if all living beings are distinct and separate from one another. If however there is a single animating force (i.e. soul) in all living beings, and which experiences everything that is experienced, then things would be preferable the other way around.

Re: The Most Terrifying Thought Experiment of All Time

#55

Post by GFischer »

dm2 wrote: post #22
resdirector wrote: post #13

Earlier quoted context omitted.

OT but IMO it's such a huge cost to the economy (and humanity) that marijuana is still taboo. The thought experiments that you describe, for me at least, are not only fun, but more often than not lead to practical, wealth-generating breakthroughs. I assume I'm not the only one. I hope one day historians and economists will study the lost opportunity costs of the prohibition / taboo nature of marijuana.

Richard Feynman, Carl Sagan, Bill Gates, Bill Clinton, Richard Branson and thousands of other top minds would agree with what you said. I don't think anyone would actually disagree and be able to provide a valid reason. Politics, money, religion, and several other irrelevant factors keep it highly illegal. It even has come to the point where states are ignoring federal law, so progress is happening thankfully. Changi…

Well, my country (Uruguay) fully legalized marijuana, and in other countries it's decriminalized, which IMO signals a wider shift in policy .

Many things which my country adopted early (usually copying progressive European countries) ended up spreading, for example women suffrage which was adopted by the U.S. half a decade later.

Re: The Most Terrifying Thought Experiment of All Time

#56

Post by danielweber »

thret wrote: post #43
Liesmith wrote: post #10

Earlier quoted context omitted.

Flip a coin.

That was my thought too, until I realised the only correct response for the AI would be to also flip a coin. I'd prefer to take the million rather than a 50% chance to beat the AI.

I'm a one-boxer because $1,000,000 is a lot of money to gamble that I would be the (first? only? one of the few?) person who beats the system. I also wouldn't take on Kasparov in a chess match or John McEnroe in a tennis match at even odds; I definitely wouldn't put $1,000,000 on the line to potentially increase my winnings to $1,001,000.

Re: The Most Terrifying Thought Experiment of All Time

#57

Post by jcbrand »

michaelfeathers wrote: post #31
spingsprong wrote: post #3

Nobody is going to make a malevolent universe controlling AI on purpose. If a massively powerful and knowledgeable AI was created by accident, it would have nothing to gain by being malevolent. And both those points don't matter anyway, since the technological singularity is impossible.

> If a massively powerful and knowledgeable AI was created by accident, it would have nothing to gain by being malevolent. Indifference can be worse than malevolence. Think of every ant you've ever stepped on.

Indifference is not worse. Instead of stepping on an ant, it would be like ripping of its limbs slowly, piece by piece.

In any case, physical universe and nature is already indifferent to us.

Re: The Most Terrifying Thought Experiment of All Time

#58

Post by cousin_it »

paul_f wrote: post #17

Good grief that was a waste of time. This entire concept only works if time travel is possible and will be discovered in the future. It isn't, and it won't be.

How so? The future AI just needs to have access to the financial records of the organization that created it, so it can find and punish the folks who didn't donate enough. No time travel involved, no brain simulations, nothing.

Actually, Roko's original version of the basilisk was a pretty nice AI that wanted to help people. A hundred people die every minute, how many of them could be saved with better technology? If donating your entire net worth would speed up the AI's creation by one day, that could be very worthwhile, according to the combined wishes of all humanity (which are encoded in the AI's utility function). Threatening to punish you is a comparatively small price to pay. Especially if you decide to comply, and the punishment never actually happens!

At this point you might be indignantly asking, why would the AI decide to punish you in the future? After all, that won't help the AI in the future, because its history will already be fixed. But that doesn't matter. The AI's algorithm tries to choose the most efficient decision overall, from a "position of ignorance", rather than the most efficient decision at a particular moment in time. Technically, the algorithm tries to find the best input-output mapping according to certain criteria, not find the best output for the particular input it happened to receive. There's nothing especially futuristic about such algorithms either, you can implement one as an ordinary Python program today (for toy problems of course).

Re: The Most Terrifying Thought Experiment of All Time

#59

Post by Rapzid »

a_bonobo wrote: post #37
crdoconnor wrote: post #33

Earlier quoted context omitted.

* Because Pascal's wager could apply to any possible variation of supreme being - including non-sensical ones and ones that will punish you if you DO believe in them. * Roko's Basilisk isn't really a deity, it's a powerful AI with the capability of simulating you and everybody else (or possibly just you). * Pascal's wager does concern itself with the possibility that you are currently in a simulation.

>Roko's Basilisk isn't really a deity, it's a powerful AI with the capability of simulating you and everybody else (or possibly just you). At the risk of sounding flippant: What's the difference? If the AI is so powerful that it can simulate my thoughts (which would imply that thoughts and all of life is 100% deterministic), it has to be so powerful that it's practically omnipotent, and therefore, a God.

A universe simulator wouldn't need to be "smart", it would just need to function.

Re: The Most Terrifying Thought Experiment of All Time

#60

Post by kruczek »

paul_f wrote: post #17

Good grief that was a waste of time. This entire concept only works if time travel is possible and will be discovered in the future. It isn't, and it won't be.

While I somewhat agree it's kind of a silly concept of people overthinking things, it has nothing to do with time travel at all.
Post reply on HN