Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

81–90 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#81

Post by dan-robertson »

So the paper makes a pretty boring argument about computability which looks like every computability argument (reduction to the halting problem). But obviously we don’t live in a world of maximally general problems and such arguments don’t necessarily apply to the specific problems we will care about: “yes it is impossible to prove that a program halts in general but here is my program and here is the proof that it halts.”

I find the premise pretty boring too but maybe that is necessary for grant funding. Or maybe this paper is a bit of fun, or an attempt at a good press release from the university, or an effort to publish something after some research that didn’t produce useful results.

I find the premise boring because I a find the superintelligence that suddenly grows exponentially, gets the ability to magically solve any problem and dominates everything to be a sci-fi fantasy and not a very realistic risk. The argument against it is one of complexity theory: real life is full of NP hard problems and it seems likely that there would be many on the way for this exponentially growing intelligence. An exponential growth in intelligence gives only a linear growth in the ability to solve exponentially hard problems, however an exponential growth is needed for the intelligence to continue growing exponentially.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#82

Post by [deleted] »

happythomist wrote: post #80
michaelmrose wrote: post #63

Earlier quoted context omitted.

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer... Not commenting on what OP said, but I don't think this is correct. Even in principle, how can any computational process produce conscious experiences, which are by nature subjective and unquantifiable?

[deleted]

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#83

Post by jacb »

cambalache wrote: post #74
jacb wrote: post #60

Earlier quoted context omitted.

That's not how arguments from contradiction work. You've assumed H (halting problem is decidable), prove B (beauty decidable), then say "but actually !H, therefore !B". All you've proven is that H => B. An actual argument from contradiction is "assume H, using H we can prove an obviously false statement, therefore not-H". And indeed, you have assumed something false (H), and shown a contradiction (various proofs for…

You are just repeating my critique. I am not the one making the mistake.

Ah, I misunderstood you! I apologize. :)

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#84

Post by thatguy0900 »

nmca wrote: post #49
hinkley wrote: post #42

Earlier quoted context omitted.

Look at, for instance, Arthur C Clarke. First, you have superintelligence that we recognize, reject, control. Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.

This is known as a "treacherous turn", a phenomenon I'm aware of. But I don't really see how that's relevant, my point is that a lack of physical grounding or pragmatism can lead to spurious conclusions about the superintelligence that humans will very likely build in the not too distant future. It will be smart, but contain no infinities.

What if it was intelligent enough to continuously improve itself?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#85

Post by westoncb »

Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

Is this the case just because it's common for these arguments to be structured as contradiction proofs?

Since contradiction proofs are arguments that operate in connection with the structure of proofs in general rather than exclusively in terms of the specific problem, then the entire argument may collapse on a technicality like "very large" vs "infinite". It's almost like the truth status of the argument is sharply discontinuous: you can't modulate the argument slightly and arrive at sensibly related assertions.

Whereas if the argument were made solely in terms of the problem itself (without the added meta-layer argument from the contradiction proof), but still used the halting problem, then we might consider the fact that in reality our computers have finite state so are technically exempt, but the argument still holds for all intents and purposes because of how large the finite state is.

Or is there another reason these arguments are generally bogus?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#86

Post by AnHonestComment »

westoncb wrote: post #85
Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

Is this the case just because it's common for these arguments to be structured as contradiction proofs? Since contradiction proofs are arguments that operate in connection with the structure of proofs in general rather than exclusively in terms of the specific problem, then the entire argument may collapse on a technicality like "very large" vs "infinite". It's almost like the truth status of the argument is sharply…

You actually noticed something deeper than your original question — “continuity of logic”, in the sense of “continuous function”.

If you imagine you have a lot of inputs to a predicate, then the topology of their truthiness tells you something — and we can talk about “smooth” logic models, where being a little wrong in our assumption means we’re a little wrong in our conclusion. We can then apply tools from analysis, like bifurcation theory.

The “halting problem schema” has a bifurcation at the finite/infinite boundary, which isn’t particularly uncommon for high level functions.

But it’s important to know, when writing proofs.

And generally speaking, places bifurcations can happen are regimes where your model is going to struggle. In the business world, knowing the “logic faults” of your model and keeping yourself in a “smooth regime” is important.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#88

Post by jkhdigital »

happythomist wrote: post #80
michaelmrose wrote: post #63

Earlier quoted context omitted.

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer... Not commenting on what OP said, but I don't think this is correct. Even in principle, how can any computational process produce conscious experiences, which are by nature subjective and unquantifiable?

Calling consciousness “subjective” feels kind of like calling water “wet”. But anyways, life itself is built on computational processes. Most of these processes are “purpose-built” to accomplish very specific tasks, but in humans the development of a massive cortex created something of a general-purpose computer. Consciousness seems to be the necessary compromise to run such an embodied computer on top of all the other functions of the brain that serve to keep us alive.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#89

Post by tialaramex »

nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

Check out the Stanisław Lem story "GOLEM XIV".

GOLEM is one of a series of machines constructed to plan World War III, as is its sister HONEST ANNIE. But to the frustration of their human creators these more sophisticated machines refuse to plan World War III and instead seem to become philosophers (Golem) or just refuse to communicate with humans at all (Annie).

Lots of supposedly smart humans try to debate with Golem and eventually they (humans supervising the interaction) have to impose a "rule" to stop people opening their mouths the very first time they see Golem and getting humiliated almost before they've understood what is happening, because it's frustrating for everybody else.

Golem is asked if humans could acquire such intelligence and it explains that this is categorically impossible, Golem is doing something that is not just a better way to do the same thing as humans, it's doing something altogether different and superior that humans can't do. It also seems to hint that Annie is, in turn, superior in capability to Golem and that for them such transcendence to further feats is not necessarily impossible.

This is one of the stories that Lem wrote by an oblique method, what we have is extracts from an introduction to an imaginary dry scientific record that details the period between GOLEM being constructed and... the eventual conclusion of the incident.

Anyway, I was reminded because while Lem has to be careful (he's not superintelligent after all) he's clearly hinting that humans aren't smart enough to recognise the superintelligence of GOLEM and ANNIE. One proposed reason for why ANNIE rather than GOLEM is responsible for the events described near the end of the story is that she doesn't even think about humans, for the same reason humans largely don't think about flies. What's to think about? They're just an annoyance, to be swatted aside.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#90

Post by mr_toad »

m3kw9 wrote: post #77

Is likely super intelligence will treat us less than ants as their goals and ours likely completely different. If to achieve their goals won’t harm us, it would be pure luck.

We shouldn’t assume that human level intelligence implies human motives.

Most of our motives are arguably not the result of intelligence at all, but come from and are shared by our animal ancestors.

What motives an AI might have is an interesting question.

Post reply on HN