Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

61–70 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#61

Post by hansvm »

xt00 wrote: post #12

Currently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could the super intelligence give rise to something more intelligent than it? The answer must be yes, then the question is if containment is the problem and the conclusion is that it cannot be contained, then what we should be making right now is a super intelligen…

Even if the hypotheses hold, a sequence x, f(x), f(f(x)), ... doesn't have to diverge even with f(x)>x for all x. A chain of super intelligences could feasibly have a small upper bound not much greater than that for a single person.

That idea even lines up with some crude experimental data -- vast additional resources barely make a dent in the frontier for chess, image recognition, a breadth of graph problems, etc....

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#62

Post by [deleted] »

jacb wrote: post #60
cambalache wrote: post #53

I confess I am a frustrated academic, but from time to time I read a paper like this one and and convince myself that my frustration may perhaps be unwarranted. RECIPE TO PROVE ANYTHING IS INCOMPUTABLE (According to this paper). Example, beauty is incomputable Step 1. Assume there is an algorithm Beauty(R,D) that given the program R and the input D, will return True if it is beautiful and False otherwise. Step 2. Cre…

That's not how arguments from contradiction work. You've assumed H (halting problem is decidable), prove B (beauty decidable), then say "but actually !H, therefore !B". All you've proven is that H => B. An actual argument from contradiction is "assume H, using H we can prove an obviously false statement, therefore not-H". And indeed, you have assumed something false (H), and shown a contradiction (various proofs for…

[deleted]

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#63

Post by michaelmrose »

djkndjn wrote: post #45
Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

what you need to understand about the halting problem is that at is core it is an epistemological problem. An other expresion of it are Gödel's incompleteness theorems. Imagine a blank sheet of paper the area of the paper is what is knowable now start building logic as a data structure. We start with the first nodes which are the axioms now everything derived conects to other nodes etc as it expands as mold its going…

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat.

Current AI which probably doesn't fit either of our definitions of intelligence can design novel untrained strategies. See for example AlphaGo Zero.

It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer but more broadly a computer doesn't necessarily mean a silicon chip any more than it meant a vacuum tube. If silicon chips prove an insufficient mechanism there is no particular reason we couldn't somehow use a biological substrate or indeed even one we haven't thought of.

Perhaps you simply need to acknowledge our limited understanding of the brain AND expand your definition of the term computer.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#64

Post by biolurker1 »

strofcon wrote: post #7
biolurker1 wrote: post #6

Earlier quoted context omitted.

Until it isn't and we are faced yet again with a much worse pandemic like situation.

Except that actual pandemics are demonstrable, predictable, and based in known science, yes?

Superintelligence is a mathematical proof

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#65

Post by biolurker1 »

kryptiskt wrote: post #36
biolurker1 wrote: post #6

Earlier quoted context omitted.

Until it isn't and we are faced yet again with a much worse pandemic like situation.

Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world The visible universe is far too small to store that data. Like many exponentials too small. You can't even enumerate all the programs that work on a handful of inputs without running into the problem that the universe just…

Until there is a breakthrough. World genius were able to advanced humanity because they were open to possibilities. Lesser scientists were dogmatic that "this can't be because so and so"

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#66

Post by plutonorm »

semi-extrinsic wrote: post #41
plutonorm wrote: post #32

Earlier quoted context omitted.

That's the mainstream opinion on every. single. revolutionary advance. That you and everyone else believes it's not going to happen ever has almost no predictive power as to whether it actually will.

It's not so much "opinion on a revolutionary advance". When it comes to AGI-related stuff, we are quite literally like contemporaries of Leonardo da Vinci, who have seen his plans for the helicopter and are postulating that helicopters will cause big problems if they fly too high and crash into the mechanism that is holding up the sky above us. Also, this is not the mainstream opinion on e.g. fusion, or electric cars…

Said just about everyone before the Wright brothers made their first flight.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#67

Post by uyt »

ben_w wrote: post #39
Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

Different for pure mathematics sure, but is that of practical importance given how fast busy-beaver numbers grow?

I don't understand the busy beaver stuff, but is this argument similar:

On a 32-bit computer, there are 2^32 bytes of memory or 256^(2^32)=2^34359738368 possible states. A program is something that takes you from one memory state to another and you want to figure out if the state transitions form a cycle?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#68

Post by yongjik »

nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

Heh, that quote reeks of postmodernism - reminds me of the whole Sokal affair. Good old days...

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#69

Post by boxmonster »

tracedddd wrote: post #48
boxmonster wrote: post #43

These arguments about hypothetical super intelligences are interesting, but my concern is not very great because we can just pull the power plug if necessary

there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.

> You’d have to pull the plug on _everything_ to shut it down.

Right. Individual countries have shut off their internet. Why not the world?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#70

Post by boxmonster »

twhitmore wrote: post #55
boxmonster wrote: post #43

These arguments about hypothetical super intelligences are interesting, but my concern is not very great because we can just pull the power plug if necessary

I think this assumption is worth at least casually testing. Turning off sounds easy -- Except that it's almost certainly connected to, or adjacent to, the network. Where else are the large volumes of training data coming from? Presumably being super-intelligent the AI would be: 1) likely to anticipate being turned off, 2) super-normally capable to identify & traverse any network security measures; 3) potentially also…

I'm confident safety measures can be developed. I worked in the nuclear field where the concept of "fail safe" is very important. I'm much more concerned about a corrupt human dictator utilizing nearly-super-intelligence to enslave people, which seems the far more likely case.
Post reply on HN