On being an AI in a box

Am I the only one who is totally unimpressed by this? So -- someone can convince someone else to let them out of a cell. Ok. Fine. There have been prison breakouts before.

But what does this have to do with a powerful AI vs. a human? Obviously Eliezer was not to his interlocutor as a human is to an animal, even if his partner was really dumb. Moreover, two trials is not a trend. This AI experiment was not a well designed experiment and did not involve an AI.

...and that "two trials" thing is not just a nitpick:

http://lesswrong.com/lw/up/shut_up_and_do_the_impossible/

So, after investigating to make sure they could afford to lose it, I played another three AI-Box experiments. I won the first, and then lost the next two. And then I called a halt to it. I didn't like the person I turned into when I started to lose.

I suspect that something with approximately human equivalent intelligence would require real world embodiment to learn and manipulate things, so it might not even be likely that a super AI could be locked in a box. I guess this embodiment could be limited to senses though (having eyes/ears.. but no arms or evolved weapons like claws etc.. a living head on a table isn't very threatening, and can receive information from the world.. it just can't send much information out to the world, so to speak, in the form of manipulating it with arms or a body).

I guess another possibility could be that there's an artificial polygon (or similar) environment inside the box with it (seems like it'd need something to interact with or intelligence would basically be meaningless and it'd be a helen keller (who couldn't touch or taste either). Maybe in the future we'll have programmed models of the real world that are nearly as rich as it (the matrix) so the robot could 'exist' there until a human decides to embody it in the real world instead. I kinda doubt an artificial world can be nearly as real as the real world though just due to computational irreducibility (for example, if you're in a polygon world and look at stuff with a microscope, you'll see pixels, not molecules). An artificial world and the real world might be too incompatible to transfer an intelligence from one to the other, so maybe the only way a super intelligence could come about is in the real, complex, atom filled world.

The singularity seems like a quasi religion and/or SEO tactic mostly by people who used to play dungeons and dragons.

It is a disproof by counterexample. People claim that if we couldn't trust a superhuman AI to be free in the world, why don't we just lock it in a box and then ask it about cures for cancer and so on? If we just ask it, and all it can do is talk what harm can it do?

They say, all we need to do is say "no" if it asks to be released.

This experiment shows that if you can't resist freeing a human locked in a box, you don't stand a chance against an AI.

How do you think Eliezer managed his original "trick"?

Oh no, somebody is doing research in an as of yet unexplored area! That must be a humongous waste of time! Pull the plug before too many papers have been written!

Before I read Eliezer's work I thought the design an AI first, think about details & security afterward was a viable strategy. Eliezer illustrates how such an approach can go very wrong. Valuable research, if you ask me.

PS: pulling the plug misses the point. The point is that a transhuman AI can convince the GK that the GK wants to let the AI go free. The AI can offer a big bribe (money/power/reputation) and argue that eventually somebody will let an AI go free anyway, so why not stop the futile resistance, accept the bribe, and move on?