Superintelligence cannot be contained: Lessons from Computability Theory • Hacker News Board

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#111

Post by dvdkhlng » Tue, Jan 12, 2021, 4:12 AM UTC

ben_w wrote: post #73

uyt wrote: post #67

Earlier quoted context omitted.

I don't understand the busy beaver stuff, but is this argument similar: On a 32-bit computer, there are 2^32 bytes of memory or 256^(2^32)=2^34359738368 possible states. A program is something that takes you from one memory state to another and you want to figure out if the state transitions form a cycle?

Similar idea, but with Turing machines rather than physical computers. Simplified a bit, the n-th busy beaver number is the number of 1s which can be written out by an n-state Turing machine. For 2-symbols and n-states, the sequence goes: 6, 21, 107, >= 47176870, > 7.4×10^36534, > 10^10^10^10^18705353 — and those inequalities are because the numbers themselves cannot be computed directly, you have to run the program…

I think this approach quickly enters the realm of transcomputational problems [1], which, given the limits of this universe is equivalent to "undecidable" for all practical considerations.

[1] https://en.wikipedia.org/wiki/Transcomputational_problem

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#112

Post by astrophysician » Tue, Jan 12, 2021, 4:23 AM UTC

bobthechef wrote: post #109

michaelmrose wrote: post #63

Earlier quoted context omitted.

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer We don't need to know. It suffices to expand "reason" to include not just inference, but properly those things which inference depends on, namely, conceptualization+abstraction and judgement. You could never infer that "all triangles have angles adding to 180 degrees" unless…

> The burden of proof is on those claiming minds and computers are the same thing.

I never understand this argument. IMHO the rest of your argument seems to obfuscate something rather simple... your brain is a physical object. The atoms that make up the neurons, glial cells, etc. are real and are governed by physics. Are you saying that what you think of as "you" is made up of more than these physical ingredients? To me that would be invoking something "magical" and would thus bear the burden of proof.

> You can have concrete green triangular objects in the world, but not triangularity or green as such...except in a mind that is able to abstract universals from particulars

Why can a sufficiently powerful computer not conceptualize these things? How is that fundamentally different than what goes on in your brain?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#113

Post by masswerk » Tue, Jan 12, 2021, 4:32 AM UTC

nmca wrote: post #40

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “har…

> because obviously we will be able to recognise it

Mind that even the oldest definitions of intelligence include the ability to assume a position, which is not necessarily held by the subject. This includes all kinds of imitation and mimicry, i.e., simulating a state. Assuming that super intelligence is an extended state of "normal" intelligence, but excelling in capabilities, we must also assume assume that we would be only able to recognize it, if the superintelligence wants to be recognized or allows for it in any other way (e.g., doesn't care about being recognized, etc). Now we could obviously take simulation out of the equation (or definition), but then, what are we talking about?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#114

Post by astrophysician » Tue, Jan 12, 2021, 4:44 AM UTC

happythomist wrote: post #108

jkhdigital wrote: post #88

Earlier quoted context omitted.

Calling consciousness “subjective” feels kind of like calling water “wet”. But anyways, life itself is built on computational processes. Most of these processes are “purpose-built” to accomplish very specific tasks, but in humans the development of a massive cortex created something of a general-purpose computer. Consciousness seems to be the necessary compromise to run such an embodied computer on top of all the oth…

I don't think you can say that life is built on computational processes unless you use a definition of "computation" that is so vague and all-encompassing that it becomes effectively meaningless. The Wikipedia definition of "computation" is "any type of calculation that includes both arithmetical and non-arithmetical steps and which follows a well-defined model". But this only makes sense in the context of a designer…

> I don't think you can say that life is built on computational processes unless you use a definition of "computation" that is so vague and all-encompassing that it becomes effectively meaningless.

I mean, there's nothing physically stopping us from simulating a brain, right? It's a finite object with a finite amount of physical ingredients, and therefore with a finite amount of computing power we can simulate what it does. To me personally, that's a computational process. Maybe that's an overly broad definition of computation, but I think these debates tend to be about whether there is something fundamentally different about "life" (by which I assume you include consciousness). But maybe that's not what you're saying.

> He points out that "if we are to suppose that the brain is a digital computer, we are still faced with the question 'And who is the user?'"

What does that question even mean? I think it seems deep because we humans have a tendency to ascribe some sort of supernatural aura to our lived experience. Life is something incredible but that (at least to my knowledge) is not uncomputable...

> There is no computational process that will produce the sensations of colour or sound or touch.

Got one: the brain!

> At best you will have some representation that requires having actually experienced those sensations to understand it.

Why does a computer not "experience" something?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#115

Post by michaelmrose » Tue, Jan 12, 2021, 5:03 AM UTC

bobthechef wrote: post #109

michaelmrose wrote: post #63

Earlier quoted context omitted.

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer We don't need to know. It suffices to expand "reason" to include not just inference, but properly those things which inference depends on, namely, conceptualization+abstraction and judgement. You could never infer that "all triangles have angles adding to 180 degrees" unless…

The burden of proof is on you to prove that a "mind" that is meaningfully different from a very powerful computer exists in the first place in order for anyone to entertain the idea of whether not a computer can approximate it.

At first blush the mind is just an emergent property of the powerful biological computer in your skull. If you think its animated by a spiritual component its on you to provide that that even exists.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#116

Post by blonde_ocean » Tue, Jan 12, 2021, 5:04 AM UTC

westoncb wrote: post #85

Animats wrote: post #18

Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons…

Is this the case just because it's common for these arguments to be structured as contradiction proofs? Since contradiction proofs are arguments that operate in connection with the structure of proofs in general rather than exclusively in terms of the specific problem, then the entire argument may collapse on a technicality like "very large" vs "infinite". It's almost like the truth status of the argument is sharply…

How else do you prove something using the halting problem other than a proof of contradiction?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#117

Post by happythomist » Tue, Jan 12, 2021, 5:04 AM UTC

astrophysician wrote: post #114

happythomist wrote: post #108

Earlier quoted context omitted.

I don't think you can say that life is built on computational processes unless you use a definition of "computation" that is so vague and all-encompassing that it becomes effectively meaningless. The Wikipedia definition of "computation" is "any type of calculation that includes both arithmetical and non-arithmetical steps and which follows a well-defined model". But this only makes sense in the context of a designer…

> I don't think you can say that life is built on computational processes unless you use a definition of "computation" that is so vague and all-encompassing that it becomes effectively meaningless. I mean, there's nothing physically stopping us from simulating a brain, right? It's a finite object with a finite amount of physical ingredients, and therefore with a finite amount of computing power we can simulate what i…

I think you're missing the central point, which is that computation is observer relative. Anything can be interpreted as a computational process.

Searle: "Thus for example the wall behind my back is right now implementing the Wordstar program, because there is some pattern of molecule movements which is isomorphic with the formal structure of Wordstar. But if the wall is implementing Wordstar then if it is a big enough wall it is implementing any program, including any program implemented in the brain."

That's why Searle asks "who is the user?" At some point things have to stop being observer relative and have an intrinsic meaning or essence of their own.

> Got one: the brain!

That's circular reasoning. The point is that qualia are not something which, in principle, can be the subject of computation. There is no way to represent the fullness of sensation itself, like the redness of red or the softness of silk, as information. So how can our brains be "computing" it?

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#118

Post by michaelmrose » Tue, Jan 12, 2021, 5:06 AM UTC

happythomist wrote: post #80

michaelmrose wrote: post #63

Earlier quoted context omitted.

You need to start by proving that brains and all idealized computers we can build are inherently on different levels. Intuitively it seems like intuition is just a probabilistic analysis on incomplete data. For example that shadow is "probably" a predator that may eat me or that sound is probably a prey animal I can kill and eat. Current AI which probably doesn't fit either of our definitions of intelligence can desi…

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer... Not commenting on what OP said, but I don't think this is correct. Even in principle, how can any computational process produce conscious experiences, which are by nature subjective and unquantifiable?

Trivial what you think of as subjective and unquantifiable are simply so because your brain is too complicated to be taken apart while its running to inspect it effectively with our present level of technology. A subjective experience is just how your brain models your own program in order to produce a progressively refined program that will have a higher chance of successful reproduction.

The magical thing you think is beyond comprehension just isn't real.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#119

Post by happythomist » Tue, Jan 12, 2021, 5:25 AM UTC

michaelmrose wrote: post #118

happythomist wrote: post #80

Earlier quoted context omitted.

> It's not even clear exactly how our brains work so its hard to imagine that they couldn't be implemented with a sufficiently powerful computer... Not commenting on what OP said, but I don't think this is correct. Even in principle, how can any computational process produce conscious experiences, which are by nature subjective and unquantifiable?

Trivial what you think of as subjective and unquantifiable are simply so because your brain is too complicated to be taken apart while its running to inspect it effectively with our present level of technology. A subjective experience is just how your brain models your own program in order to produce a progressively refined program that will have a higher chance of successful reproduction. The magical thing you think…

How do you go about quantifying the sensory experience of red, then? You can observe that red light has a wavelength of 620 to 750 nm, or that we've assigned it the RGB colour code of #FF0000, but neither fact actually captures or explains the sensory experience. Even trying is a fool's errand, because sensory experiences are inherently qualitative, not quantitative.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#120

Post by michaelmrose » Tue, Jan 12, 2021, 5:31 AM UTC

happythomist wrote: post #108

jkhdigital wrote: post #88

Earlier quoted context omitted.

Calling consciousness “subjective” feels kind of like calling water “wet”. But anyways, life itself is built on computational processes. Most of these processes are “purpose-built” to accomplish very specific tasks, but in humans the development of a massive cortex created something of a general-purpose computer. Consciousness seems to be the necessary compromise to run such an embodied computer on top of all the oth…

I don't think you can say that life is built on computational processes unless you use a definition of "computation" that is so vague and all-encompassing that it becomes effectively meaningless. The Wikipedia definition of "computation" is "any type of calculation that includes both arithmetical and non-arithmetical steps and which follows a well-defined model". But this only makes sense in the context of a designer…

In this case let us substitute the narrow definition of thought and computation by an imitation human as a process which given the current state of the brain/computer and the current state of the universe induces changes in the brain so as to model the state of the universe both now and in response to a hypothetical possible pool of actions such that the actions of model and world become entwined in a way that could be modeled from the inside as the world being the result of choices and from the outside as choices being the result of the world.

This is true even of a chess program that attempts to model the current and possible states to the chess board in a fashion as to bring about a goal by way of selection of moves.

Suppose we take a very precise process and produce an exact physical copy of you. For being artificial it ought to experience the same sorts of experiences as you. The same ought to be true of a computer simulation of same. The same ought to be true for a variety of increasingly large modifications of the original design. After all if billions of humans can pop out divergent versions of humans who are all conscious it seems hard to argue that you are a unique configuration. In fact if we imagine working for the next 1000 years on producing a better human being that we ought to be able to produce beings who no longer regard us as truly human because we lack both subjective experiences they regard as essential and computational capability. Maybe they can hold a million times more data in their head at once and they regard us as squirrels.

These beings might regard our workings as completely explicable and replicatable in many substrates while regarding their own workings at the far limit of their own understanding as inherently beyond all possible understanding.

Both you and they are probably wrong. Searle was an asshole.