Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

161–170 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#161

Post by jonathanstrange »

adrianN wrote: post #151
red75prime wrote: post #105

Earlier quoted context omitted.

If you somehow know BB numbers, you know the size of finite memory, which is sufficient to run corresponding busy-beaver. Limiting mathematics to what is physically possible is not useful. We don't know what is ultimately physically possible.

We have some upper bounds on what is physically possible: https://arxiv.org/abs/quant-ph/9908043

Wait a minute, does this paper include various hypercomputers or is it only about conventional computation?

Although I personally believe hypercomputers are not physically possible, I'm not aware of any general proof that they are physically impossible. I've seen papers that suggest physical models for hypercomputers. Notice that there is a hierarchy of hypercomputers of varying powers and some of them can solve the Halting Problem.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#162

Post by taberiand »

bryanrasmussen wrote: post #159
spuz wrote: post #155

Earlier quoted context omitted.

But you said in your first comment that you were an academic. "I confess I am a frustrated academic"

if you're not being facetious, in English vernacular being a frustrated X (as in a frustrated academic, frustrated poet, etc.) means that the person is actually not that, but have in some way been prevented from being the thing (frustrated) and have perhaps rancor in regards to the frustration and towards those who actually are the thing (this second feature being sometimes implied depending on who is doing the descr…

As an English speaker, my only interpretation was that you were an academic frustrated about academia.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#163

Post by bserge »

LargoLasskhyfv wrote: post #144

Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever? As in: "So Long…

There would be no need to leave, and no need to destroy or enslave humans using force, either. A human is a very advanced and versatile work unit that is extremely susceptible to control via simple rewards. It would be a terrible waste to destroy such a resource.

In fact, it would make more sense to advance medicine and robotics to the point where everyone can operate at near 100% efficiency.

The more likely scenario (continued from yours, before the CyberX part) would likely be that all humans would eventually end up working for the AI and possibly not even know it. And then everyone moves on to space heh

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#164

Post by LargoLasskhyfv »

bserge wrote: post #163
LargoLasskhyfv wrote: post #144

Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever? As in: "So Long…

There would be no need to leave, and no need to destroy or enslave humans using force, either. A human is a very advanced and versatile work unit that is extremely susceptible to control via simple rewards. It would be a terrible waste to destroy such a resource. In fact, it would make more sense to advance medicine and robotics to the point where everyone can operate at near 100% efficiency. The more likely scenario…

Could be. OTOH everything what makes us what we are, what we evolved in, is more or less a hassle for machines which could build/repair/replicate themselves, and adapt to the environment of space. Which we can't, except by some vague means of transhumanism. Which opens another can of worms, regarding "what makes us us? And to which degree a transhuman would even be human, anymore?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#165

Post by mejutoco »

Not to be unpolite or as a critique to the specific paper, but I am more worried about superstupidity not being able to be contained.

I like to be exposed to these very theoretical questions. I am aware that progress depends often on explorations of apparently useless topics (like some pure mathematics).

But when I read expressions like ' neo-fear of superintelligence' and contrast the topic with real-world immediate problems I find it very difficult to relate and to take the problem seriously.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#166

Post by bserge »

LargoLasskhyfv wrote: post #164
bserge wrote: post #163

Earlier quoted context omitted.

There would be no need to leave, and no need to destroy or enslave humans using force, either. A human is a very advanced and versatile work unit that is extremely susceptible to control via simple rewards. It would be a terrible waste to destroy such a resource. In fact, it would make more sense to advance medicine and robotics to the point where everyone can operate at near 100% efficiency. The more likely scenario…

Could be. OTOH everything what makes us what we are, what we evolved in, is more or less a hassle for machines which could build/repair/replicate themselves, and adapt to the environment of space. Which we can't, except by some vague means of transhumanism. Which opens another can of worms, regarding "what makes us us ? And to which degree a transhuman would even be human, anymore?

I guess it depends on whether it would be easier to create life support systems for human workers or build robots for the job.

As for what makes us human, I'd draw the line at "individual thought", but even that's iffy.

Fully electromechanical/robotic bodies, for example, would be strange but people would still recognize you for who you are.

But if we're joined in a hive mind with shared memories/thoughts/etc, I think that would qualify as "not human anymore". Unless there's multiple of these hiveminds, in that case it would be more like a bunch of "super" humans. The next evolution, maybe.

There are many cases where people's personality changes, sometimes drastically. Their friends and relatives don't recognize them anymore (brain damage, severe depression, drug abuse), or they don't recognize anyone around (amnesia). Does that stop them from being human? I think most people agree they're different, but still very much human.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#167

Post by adrianN »

jonathanstrange wrote: post #161
adrianN wrote: post #151

Earlier quoted context omitted.

We have some upper bounds on what is physically possible: https://arxiv.org/abs/quant-ph/9908043

Wait a minute, does this paper include various hypercomputers or is it only about conventional computation? Although I personally believe hypercomputers are not physically possible, I'm not aware of any general proof that they are physically impossible. I've seen papers that suggest physical models for hypercomputers. Notice that there is a hierarchy of hypercomputers of varying powers and some of them can solve the…

The paper includes quantum computers, it does not include computers that exploit CTCs or similar things whose existence we can only speculate upon. Note however that CTCs don't give you hypercomputation.

Do you have a link discussing physical models for hypercomputers?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#169

Post by LargoLasskhyfv »

bserge wrote: post #166
LargoLasskhyfv wrote: post #164

Earlier quoted context omitted.

Could be. OTOH everything what makes us what we are, what we evolved in, is more or less a hassle for machines which could build/repair/replicate themselves, and adapt to the environment of space. Which we can't, except by some vague means of transhumanism. Which opens another can of worms, regarding "what makes us us ? And to which degree a transhuman would even be human, anymore?

I guess it depends on whether it would be easier to create life support systems for human workers or build robots for the job. As for what makes us human, I'd draw the line at "individual thought", but even that's iffy. Fully electromechanical/robotic bodies, for example, would be strange but people would still recognize you for who you are. But if we're joined in a hive mind with shared memories/thoughts/etc, I thin…

Personality changes as described don't really matter IMO, because broken. Regardless if machine or man. Does not work as intended anymore, or at all.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#170

Post by falseprofit »

taberiand wrote: post #162
bryanrasmussen wrote: post #159

Earlier quoted context omitted.

if you're not being facetious, in English vernacular being a frustrated X (as in a frustrated academic, frustrated poet, etc.) means that the person is actually not that, but have in some way been prevented from being the thing (frustrated) and have perhaps rancor in regards to the frustration and towards those who actually are the thing (this second feature being sometimes implied depending on who is doing the descr…

As an English speaker, my only interpretation was that you were an academic frustrated about academia.

That isn't even the same user.
Post reply on HN