Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

91–100 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#91

Post by cma »

nmca wrote: post #49
hinkley wrote: post #42

Earlier quoted context omitted.

Look at, for instance, Arthur C Clarke. First, you have superintelligence that we recognize, reject, control. Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.

This is known as a "treacherous turn", a phenomenon I'm aware of. But I don't really see how that's relevant, my point is that a lack of physical grounding or pragmatism can lead to spurious conclusions about the superintelligence that humans will very likely build in the not too distant future. It will be smart, but contain no infinities.

> It will be smart, but contain no infinities.

We think we can say that from physics, but we don't know what we don't know.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#92

Post by alisonkisk »

TheOtherHobbes wrote: post #46
WJW wrote: post #24

Earlier quoted context omitted.

> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.

For human-recognisable values of "resources." I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.

That's not what Dunning-Kruger means.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#93

Post by alisonkisk »

plutonorm wrote: post #66
semi-extrinsic wrote: post #41

Earlier quoted context omitted.

It's not so much "opinion on a revolutionary advance". When it comes to AGI-related stuff, we are quite literally like contemporaries of Leonardo da Vinci, who have seen his plans for the helicopter and are postulating that helicopters will cause big problems if they fly too high and crash into the mechanism that is holding up the sky above us. Also, this is not the mainstream opinion on e.g. fusion, or electric cars…

Said just about everyone before the Wright brothers made their first flight.

No one thought that flight was impossible. Birds were known in the 19th Century

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#94

Post by dragonwriter »

TheOtherHobbes wrote: post #46
WJW wrote: post #24

Earlier quoted context omitted.

> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.

For human-recognisable values of "resources." I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.

> But in true Dunning-Kruger style, we don't know what we don't know

That's not D-K, but Rumsfeldian Uncertainty.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#95

Post by qwantim1 »

nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

> obviously we will be able to recognise it.

There is no basis for that assumption, as you’ve never encountered a superintelligence knowingly.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#96

Post by qwantim1 »

xt00 wrote: post #12

Currently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could the super intelligence give rise to something more intelligent than it? The answer must be yes, then the question is if containment is the problem and the conclusion is that it cannot be contained, then what we should be making right now is a super intelligen…

The most obvious solution to problems arising from assisting the creation of a superintelligence is not to assist the creation of a superintelligence.

Unfortunately, we didn’t learn from the proliferation of nuclear weapons:

“If we don’t make nuclear weapons, others will make them. If we make them then they will make them. If we make them and they make them, we’ll have to make more. If they make them and we make them, they’ll make more...”

Note: This fallacy also helps sell guns and martial arts lessons.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#97

Post by alisonkisk »

boxmonster wrote: post #70
twhitmore wrote: post #55

Earlier quoted context omitted.

I think this assumption is worth at least casually testing. Turning off sounds easy -- Except that it's almost certainly connected to, or adjacent to, the network. Where else are the large volumes of training data coming from? Presumably being super-intelligent the AI would be: 1) likely to anticipate being turned off, 2) super-normally capable to identify & traverse any network security measures; 3) potentially also…

I'm confident safety measures can be developed. I worked in the nuclear field where the concept of "fail safe" is very important. I'm much more concerned about a corrupt human dictator utilizing nearly-super-intelligence to enslave people, which seems the far more likely case.

Is it possible for a nuclear power plant to fail catastrophically, at least in theory? More than three times?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#98

Post by gbjw »

cambalache wrote: post #74
jacb wrote: post #60

Earlier quoted context omitted.

That's not how arguments from contradiction work. You've assumed H (halting problem is decidable), prove B (beauty decidable), then say "but actually !H, therefore !B". All you've proven is that H => B. An actual argument from contradiction is "assume H, using H we can prove an obviously false statement, therefore not-H". And indeed, you have assumed something false (H), and shown a contradiction (various proofs for…

You are just repeating my critique. I am not the one making the mistake.

So you mean your frustration is warranted then?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#99

Post by AnHonestComment »

happythomist wrote: post #80
michaelmrose wrote: post #63

Earlier quoted context omitted.

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer... Not commenting on what OP said, but I don't think this is correct. Even in principle, how can any computational process produce conscious experiences, which are by nature subjective and unquantifiable?

I always find this a bit like “do submarines swim?”

Dunno, don’t care, just want a functional machine. If it gets from A to B, “swimming” is irrelevant.

————

As a matter of philosophy, the typical response is qualia happens to everything, but we only recognize it in things with dense computation and self-awareness similar to ours.

Like a cat or a dog or a whale.

We can maybe see hints of it in birds, insects (colonies), etc.

We’re starting to discover some of the complex signaling pathways in, eg, old growth forests — but they’re so out of scale and unlike us, we have trouble comprehending whatever experience, eg, the giant fungus under Oregon might have.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#100

Post by Causality1 »

tialaramex wrote: post #89
nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

Check out the Stanisław Lem story "GOLEM XIV". GOLEM is one of a series of machines constructed to plan World War III, as is its sister HONEST ANNIE. But to the frustration of their human creators these more sophisticated machines refuse to plan World War III and instead seem to become philosophers (Golem) or just refuse to communicate with humans at all (Annie). Lots of supposedly smart humans try to debate with Gol…

That relies on the existence of systems too complex for humans to understand. We may eventually build machines that can in turn build other incomprehensible machines, but we haven't passed that point yet. Superintelligence can't surprise us until some point after computers become self-improving.
Post reply on HN