Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

11–20 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#11

Post by rhn_mk1 »

strofcon wrote: post #7
biolurker1 wrote: post #6

Earlier quoted context omitted.

Until it isn't and we are faced yet again with a much worse pandemic like situation.

Except that actual pandemics are demonstrable, predictable, and based in known science, yes?

The difference between pandemics and this is that pandemics have happened before. This is more like global warming in that respect.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#12

Post by xt00 »

Currently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could the super intelligence give rise to something more intelligent than it? The answer must be yes, then the question is if containment is the problem and the conclusion is that it cannot be contained, then what we should be making right now is a super intelligence whose sole job in life is to contain superintelligences. Which sounds problematic because containment could result in physical destruction to create the containment. Hmm... superintelligence feels an awful lot like the worst case definition of pandora's box..

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#13

Post by jandrese »

ReadEvalPost wrote: post #8
HenryKissinger wrote: post #4

> Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world Rest easy folks. This is purely theoretical.

"Purely theoretical" is too weak. "Physically impossible" is better. AI Safety guys really like using the physically impossible to advance their arguments. It's bizarre! Pure fantasy isn't something worth reacting to.

It's the classic case of seeing a curve pointing upwards and thinking it will continue doing that forever, even though the universe is like 0 for nearly-uncountable in cases where that has held true indefinitely. Every growth curve is an S curve.

The AI Singularity is like the Grey Goo scenario. An oversimplified model projected out so far in the future that its flaws become apparent.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#14

Post by vsareto »

strofcon wrote: post #7
biolurker1 wrote: post #6

Earlier quoted context omitted.

Until it isn't and we are faced yet again with a much worse pandemic like situation.

Except that actual pandemics are demonstrable, predictable, and based in known science, yes?

More like it's the politics that really decides what changes, regardless of what science proves

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#16

Post by anateus »

If you define "containment" as "provable non-harm" then sure. But there are essentially no complex physical systems that we can put such computational bounds on. Since "harm" comes in some form of physical actuation, I would argue that we can only ever get to something like the sort of confidence we can have that a particular manufactured part would succeed under load. The map is not the territory, and any computation that does not include computing the whole universe is necessarily but a map.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#18

Post by Animats »

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here.

Not being able to predict something for combinatorial or noise reasons is a more credible argument.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#19

Post by postalrat »

ReadEvalPost wrote: post #8
HenryKissinger wrote: post #4

> Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world Rest easy folks. This is purely theoretical.

"Purely theoretical" is too weak. "Physically impossible" is better. AI Safety guys really like using the physically impossible to advance their arguments. It's bizarre! Pure fantasy isn't something worth reacting to.

"Circle" and "exponential growth" are also physically impossible yet useful ideas.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#20

Post by Veedrac »

Obviously it's impossible to prove definitive statements about every possible potential action, as as per the halting problem some of those actions are unprovable.

It is as ridiculous to suggest that this means you can't contain a superintelligence as it is to suggest it means you can't, I don't know, go buy bread. In both cases you could analyze running a program that doesn't halt but you can't prove doesn't halt, and lock up your reasoning algorithm. The sensible thing is to not do that.

Post reply on HN