Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

71–80 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#71

Post by contravariant »

xt00 wrote: post #57
WJW wrote: post #24

Earlier quoted context omitted.

> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.

Definitely feels like a challenging philosophical question -- if you have a superintelligence how much more room is there above it in intelligence and what does that additional intelligence buy you?

The challenges don't even start there. The real problem is how to even determine superintelligence. If it's just a system that can do stuff a human can't then we've long since passed it. If it's a system that can achieve stuff humans can't, through whatever means, then it's necessarily beyond human intelligence to build it, at least intentionally.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#72

Post by qayxc »

1970-01-01 wrote: post #50

Perhaps not: The authors omit how planet-wide extinction scenarios would play-out for artificial life. For example, a Carrington Event would do a great deal of "containment" to AI.

A superintelligence wouldn't be bothered by it - shielding is both possible and practical.

Plus nothing would be able to stop the AI from controlling the Sun to prevent such events - it's a superintelligence after all, so I'd expect it to consider such case.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#73

Post by ben_w »

uyt wrote: post #67
ben_w wrote: post #39

Earlier quoted context omitted.

Different for pure mathematics sure, but is that of practical importance given how fast busy-beaver numbers grow?

I don't understand the busy beaver stuff, but is this argument similar: On a 32-bit computer, there are 2^32 bytes of memory or 256^(2^32)=2^34359738368 possible states. A program is something that takes you from one memory state to another and you want to figure out if the state transitions form a cycle?

Similar idea, but with Turing machines rather than physical computers.

Simplified a bit, the n-th busy beaver number is the number of 1s which can be written out by an n-state Turing machine.

For 2-symbols and n-states, the sequence goes: 6, 21, 107, >= 47176870, > 7.4×10^36534, > 10^10^10^10^18705353 — and those inequalities are because the numbers themselves cannot be computed directly, you have to run the program to see what it does.

https://en.wikipedia.org/wiki/Busy_beaver

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#74

Post by cambalache »

jacb wrote: post #60
cambalache wrote: post #53

I confess I am a frustrated academic, but from time to time I read a paper like this one and and convince myself that my frustration may perhaps be unwarranted. RECIPE TO PROVE ANYTHING IS INCOMPUTABLE (According to this paper). Example, beauty is incomputable Step 1. Assume there is an algorithm Beauty(R,D) that given the program R and the input D, will return True if it is beautiful and False otherwise. Step 2. Cre…

That's not how arguments from contradiction work. You've assumed H (halting problem is decidable), prove B (beauty decidable), then say "but actually !H, therefore !B". All you've proven is that H => B. An actual argument from contradiction is "assume H, using H we can prove an obviously false statement, therefore not-H". And indeed, you have assumed something false (H), and shown a contradiction (various proofs for…

You are just repeating my critique. I am not the one making the mistake.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#75

Post by TeMPOraL »

twic wrote: post #28
AnimalMuppet wrote: post #23

Earlier quoted context omitted.

That's easy. You set up a PhD program...

Ah! Academia is the containment mechanism!

Yup. Publish or perish is a denial-of-service scheme that slows down recursive self-improvement to the levels deemed safe for society.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#76

Post by qayxc »

stephc_int13 wrote: post #52

In my opinion, all the talks about the potential danger of advanced AI is highly speculative and revolves around a very simple thing: fear of the unknown, that's all. We simply don't know. And some people are also afraid of creation by accident, because intelligence is seen as an emergent property of complex networks, but again, this is because we don't understand much about it. Tldr; Nothing to see here, move along.

The fears and warnings of the AI safety guys are real and closer than you think.

Machine superintelligence is a bit on the extreme side, but formulating safety protocols for autonomous machines is a real challenge. We do know that optimisation functions can indeed create harmful and outright dangerous results.

It's not an unknown that AI safety researchers fear, it's actual outcomes or real experiments extrapolated into the future. Optimisation algorithms are impartial to humans, their moral code and -survival.

Simple (and often quoted) example to illustrate this point: say we create an autonomous machine that we want to help us with our stamp collection. We run an optimising algorithm and naively use the amount of unique and rare stamps collected as our target function. The loss function is determined by time taken and money spent.

Possible outcomes: the algorithm figures that the most effective way of achieving this is to hack a number of bank accounts to quickly get enough money to buy a nice collection.

Now that's bad, but maybe something we can incorporate in our training procedure: only use the money given to you.

Another possible outcome: the machine creates a set of robots that roam the planet and steal all the stamps from all over the world.

Again, you'd have to consider that in your training method: no stealing.

But in the end, you cannot foresee every possible outcome, especially since you expect the machine to come up with an unconventional solution, since otherwise you wouldn't need it in the first place.

Restricting the space of possible solutions to safe and desired ones, is a very hard (and potentially undecidable) problem. This paper is just another reminder that we have to be very careful lest we accidentally end human civilisation by means of AI.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#78

Post by tracedddd »

boxmonster wrote: post #69
tracedddd wrote: post #48

Earlier quoted context omitted.

there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.

> You’d have to pull the plug on _everything_ to shut it down. Right. Individual countries have shut off their internet. Why not the world?

somehow, I doubt we’ve achieved high enough unity for coordination like that.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#79

Post by tgbugs »

tantalor wrote: post #33

Article fails to reference Yudkowsky or "AI-box experiment" (2002) https://www.yudkowsky.net/singularity/aibox https://en.wikipedia.org/wiki/AI_box https://rationalwiki.org/wiki/AI-box_experiment

This is quite similar to why Christian theologians eventually had to give up the idea that doing good works could lead to salvation, because someone could force God to save them by doing good works thus limiting God's omnipotence. The protestants tried to come up with some hocus pocus about being able to know that you were one of the elect by reversing the causality here "Oh I'm not forcing God to save me, it's just an indication that he already did." It is amusing to see that it is the human that takes on the role of God in the AI box game.

This is also Hume's "is does not imply ought."

From this we conclude that only idiots get tricked by the AI in the box, because it is only possible to prove that you _are_ a superintelligence, it is impossible to prove that you are not. This is also a simple application of Russell's teapot. Now, the point of the story is pretty clearly that human beings are idiots when it comes to this kind of thing, but didn't we already know that?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#80

Post by happythomist »

michaelmrose wrote: post #63
djkndjn wrote: post #45

Earlier quoted context omitted.

what you need to understand about the halting problem is that at is core it is an epistemological problem. An other expresion of it are Gödel's incompleteness theorems. Imagine a blank sheet of paper the area of the paper is what is knowable now start building logic as a data structure. We start with the first nodes which are the axioms now everything derived conects to other nodes etc as it expands as mold its going…

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer...

Not commenting on what OP said, but I don't think this is correct. Even in principle, how can any computational process produce conscious experiences, which are by nature subjective and unquantifiable?

Post reply on HN