Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

151–160 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#151

Post by adrianN »

red75prime wrote: post #105
Retric wrote: post #102

Earlier quoted context omitted.

Busy-beaver only grows fast given unlimited memory. On a finite machine it’s limited by the total number of states the device has. Granted that can be 10^1,000,000,000,000+ these days, but it’s still limited so BB(10^10) on actual hardware = BB(10^100).

If you somehow know BB numbers, you know the size of finite memory, which is sufficient to run corresponding busy-beaver. Limiting mathematics to what is physically possible is not useful. We don't know what is ultimately physically possible.

We have some upper bounds on what is physically possible: https://arxiv.org/abs/quant-ph/9908043

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#152

Post by LargoLasskhyfv »

simonh wrote: post #149
LargoLasskhyfv wrote: post #144

Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever? As in: "So Long…

More intelligent systems will find more efficient ways to achieve their goals than dumber ones (us). They will also select more effective intermediate goals and 'stepping stone' states towards their goals than we can anticipate. The consequence of this is that we cannot anticipate what means and intermediate objectives they will use. That's exactly what you are doing in your example, but attempting to speculate on th…

We: "Send post cards, will ya?"

They/it: "Sure thing, Kittenz!"

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#153

Post by adrianN »

djkndjn wrote: post #45
Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

what you need to understand about the halting problem is that at is core it is an epistemological problem. An other expresion of it are Gödel's incompleteness theorems. Imagine a blank sheet of paper the area of the paper is what is knowable now start building logic as a data structure. We start with the first nodes which are the axioms now everything derived conects to other nodes etc as it expands as mold its going…

Your brain can store a finite amount of state because it's a finite piece of matter. You are therefore a finite automaton manipulating symbols outside of your brain, e.g. on a piece of paper. That makes you no more powerful than a Turing machine. In fact, since your lifetime is bounded, the amount of paper your can write on is bounded too. You are in thus less powerful than a Turing machine.

The fact that you can recognize some paradoxes means exactly nothing about your computational power. There is a sufficiently large lookup table that contains the correct answers to any question you might be able to read during your lifetime. Evaluating it requires nothing more than a finite automaton.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#154

Post by geocar »

tialaramex wrote: post #89
nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

Check out the Stanisław Lem story "GOLEM XIV". GOLEM is one of a series of machines constructed to plan World War III, as is its sister HONEST ANNIE. But to the frustration of their human creators these more sophisticated machines refuse to plan World War III and instead seem to become philosophers (Golem) or just refuse to communicate with humans at all (Annie). Lots of supposedly smart humans try to debate with Gol…

> for the same reason humans largely don't think about flies. What's to think about? They're just an annoyance, to be swatted aside.

We can choose to think of flies; to imagine they are as people, with their own (short) lives, wants and wishes and dreams; that there is fly art and fly culture equal to our own. We can so imbue those flies with our animism and ethics, and treat them the way we should (ethically) treat any other person.

If we can do that, I think there is a good chance our gods can do the same, but using their superintelligent-culture, and their superintelligent-art, and their superintelligent-ethics, but imagining as we can the fly who is human, they choose to imagine human who is god.

Maybe the point is our gods can be kind to us if we can find some way to convince ourselves to be kind to the fly.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#155

Post by spuz »

cambalache wrote: post #129
gbjw wrote: post #98

Earlier quoted context omitted.

So you mean your frustration is warranted then?

If that is the kind of work published by academics associated with the MIT and La Pontificia Universidad de Madrid, then maybe I should not feel frustrated after all by not being an academic.

But you said in your first comment that you were an academic.

"I confess I am a frustrated academic"

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#156

Post by spuz »

boxmonster wrote: post #69
tracedddd wrote: post #48

Earlier quoted context omitted.

there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.

> You’d have to pull the plug on _everything_ to shut it down. Right. Individual countries have shut off their internet. Why not the world?

That would require everyone in the world committing themselves to living without computers for the rest of time.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#157

Post by Caylio »

spuz wrote: post #155
cambalache wrote: post #129

Earlier quoted context omitted.

If that is the kind of work published by academics associated with the MIT and La Pontificia Universidad de Madrid, then maybe I should not feel frustrated after all by not being an academic.

But you said in your first comment that you were an academic. "I confess I am a frustrated academic"

He is proving his point by contradicting himself, a true mathematician.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#158

Post by harmoat »

Causality1 wrote: post #100
tialaramex wrote: post #89

Earlier quoted context omitted.

Check out the Stanisław Lem story "GOLEM XIV". GOLEM is one of a series of machines constructed to plan World War III, as is its sister HONEST ANNIE. But to the frustration of their human creators these more sophisticated machines refuse to plan World War III and instead seem to become philosophers (Golem) or just refuse to communicate with humans at all (Annie). Lots of supposedly smart humans try to debate with Gol…

That relies on the existence of systems too complex for humans to understand. We may eventually build machines that can in turn build other incomprehensible machines, but we haven't passed that point yet. Superintelligence can't surprise us until some point after computers become self-improving.

Aren't large neural network already black boxes we don't understand built by machines we understand?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#159

Post by bryanrasmussen »

spuz wrote: post #155
cambalache wrote: post #129

Earlier quoted context omitted.

If that is the kind of work published by academics associated with the MIT and La Pontificia Universidad de Madrid, then maybe I should not feel frustrated after all by not being an academic.

But you said in your first comment that you were an academic. "I confess I am a frustrated academic"

if you're not being facetious,

in English vernacular being a frustrated X (as in a frustrated academic, frustrated poet, etc.) means that the person is actually not that, but have in some way been prevented from being the thing (frustrated) and have perhaps rancor in regards to the frustration and towards those who actually are the thing (this second feature being sometimes implied depending on who is doing the description of the person as frustrated)

on edit: formatting, clarification

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#160

Post by bserge »

tsimionescu wrote: post #138
happythomist wrote: post #136

Earlier quoted context omitted.

> Perhaps if we were able to understand the brain's inner workings, we could see that 'the experience of red' is precisely 'these 3 neurons firing ever 0.0112 seconds at an intensity of X while receiving 0.001 micrograms of serotonin' (completely made up, obviously). Even if we knew that a person saw red when such and such neurons fired, the neurons firing would still just be a material correlate. It would be in no w…

You are thinking of something similar to the level of today's neuroscience and brain imaging, where indeed we can only establish correlations. But I am talking about a much more in-depth understanding of the working of the brain, similar to the level of understanding we have of a microprocessor all the way from transistors to the algorithms running on it. If we could understand human thought at a similar level, we MI…

> the level of understanding we have of a microprocessor all the way from transistors to the algorithms running on it

Good example, because the vast majority of people don't understand that. I tried and I still don't, nevermind someone who doesn't even care.

I mean, I know the theory, I know the individual parts, but can't quite fully understand how a complete processor works.

If someone from as early as 1920 would find an advanced robot that is a combination of some Boston Dynamics model and offline/autonomous Google Assistant (so it could walk, listen/talk/reply, and maybe pick stuff up), they would not be able to figure out how its "brain" works. At best they'd have a general idea/theory.

Same thing with our brains and current understanding of it. I believe it is possible to reverse engineer it completely, but not with today's tools.

Post reply on HN