Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

41–50 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#41

Post by semi-extrinsic »

plutonorm wrote: post #32
st1x7 wrote: post #21

This is just science fiction. To mention "recent developments" in the introduction is somewhat misleading considering how far the current state of technology is from their hypothetical superintelligence. We don't have superintelligence, we don't have the remote idea of how to get started on creating it, in all likelihood we don't even have the correct hardware for it or any idea what the correct hardware would look l…

That's the mainstream opinion on every. single. revolutionary advance. That you and everyone else believes it's not going to happen ever has almost no predictive power as to whether it actually will.

It's not so much "opinion on a revolutionary advance". When it comes to AGI-related stuff, we are quite literally like contemporaries of Leonardo da Vinci, who have seen his plans for the helicopter and are postulating that helicopters will cause big problems if they fly too high and crash into the mechanism that is holding up the sky above us.

Also, this is not the mainstream opinion on e.g. fusion, or electric cars and smartphones (20 years ago), or a computer in every home (50 years ago). Those have been arguments about money and practicality, not about "we don't even know how such technology would look or what it would be based on".

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#42

Post by hinkley »

nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

Look at, for instance, Arthur C Clarke.

First, you have superintelligence that we recognize, reject, control.

Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#44

Post by tintor »

inglor_cz wrote: post #37
EVa5I7bHFq9mnYK wrote: post #31

It's kinda obvious: if you create a system so intelligent that it can create itself, and you impose controls over it, it will be able to create a version of itself without the controls.

Well, you can, for example, try to limit its total energy budget. That is physical limitation, that is hard to circumvent. Of course, it is possible that said superintelligence develops an ingenious new source of energy as a response.

Or it could just prevent you from limiting its total energy budget.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#45

Post by djkndjn »

Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

what you need to understand about the halting problem is that at is core it is an epistemological problem. An other expresion of it are Gödel's incompleteness theorems. Imagine a blank sheet of paper the area of the paper is what is knowable now start building logic as a data structure. We start with the first nodes which are the axioms now everything derived conects to other nodes etc as it expands as mold its going to cover some of the paper but wont be able to cover all. So the danger here is that computers are living on that fractal dimension and will never be able to see outside of it but us human beings have as kurt said intuition. The fact that we can find this paradoxes in logic means that our brains operate on a higher dimension and that computers will allways have blind spots.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#46

Post by TheOtherHobbes »

WJW wrote: post #24
xt00 wrote: post #12

Currently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could the super intelligence give rise to something more intelligent than it? The answer must be yes, then the question is if containment is the problem and the conclusion is that it cannot be contained, then what we should be making right now is a super intelligen…

> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.

For human-recognisable values of "resources."

I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#47

Post by ben_w »

jandrese wrote: post #13
ReadEvalPost wrote: post #8

Earlier quoted context omitted.

"Purely theoretical" is too weak. "Physically impossible" is better. AI Safety guys really like using the physically impossible to advance their arguments. It's bizarre! Pure fantasy isn't something worth reacting to.

It's the classic case of seeing a curve pointing upwards and thinking it will continue doing that forever, even though the universe is like 0 for nearly-uncountable in cases where that has held true indefinitely. Every growth curve is an S curve. The AI Singularity is like the Grey Goo scenario. An oversimplified model projected out so far in the future that its flaws become apparent.

Depends which version of “the singularity” is being discussed. IIRC, the original (almost certainly false) idea was a sufficiently powerful AI can start a sequence of ever power powerful AI with decreasing time between each step — reaching infinite power in finite time.

I don’t need that version of the singularity to be worried.

I think in terms of “the event horizon” rather than “the singularity”: all new tech changes the world, when the rate of change exceeds our capacity to keep up with the consequences, stuff will go wrong on a large scale for lots of people.

As for grey goo? Self replicating nanomachines is just biology. It gets everywhere, and even single-celled forms can kill you by eating your flesh or suborning your cells, but it’s mostly no big deal because you evolved to deal with that threat.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#48

Post by tracedddd »

boxmonster wrote: post #43

These arguments about hypothetical super intelligences are interesting, but my concern is not very great because we can just pull the power plug if necessary

there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#49

Post by nmca »

hinkley wrote: post #42
nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

Look at, for instance, Arthur C Clarke. First, you have superintelligence that we recognize, reject, control. Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.

This is known as a "treacherous turn", a phenomenon I'm aware of. But I don't really see how that's relevant, my point is that a lack of physical grounding or pragmatism can lead to spurious conclusions about the superintelligence that humans will very likely build in the not too distant future. It will be smart, but contain no infinities.
Post reply on HN