Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

171–180 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#171

Post by bserge »

LargoLasskhyfv wrote: post #169
bserge wrote: post #166

Earlier quoted context omitted.

I guess it depends on whether it would be easier to create life support systems for human workers or build robots for the job. As for what makes us human, I'd draw the line at "individual thought", but even that's iffy. Fully electromechanical/robotic bodies, for example, would be strange but people would still recognize you for who you are. But if we're joined in a hive mind with shared memories/thoughts/etc, I thin…

Personality changes as described don't really matter IMO, because broken . Regardless if machine or man. Does not work as intended anymore, or at all.

Well then you'd need to define "normal" and that's not easy because everyone is different. It can even be dangerous. Easy example: some people (still) define having a different religion or being gay as "broken".

People that go through depression, suicide thoughts/attempts and then recover are also different, because they gained new information.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#172

Post by bryanrasmussen »

taberiand wrote: post #162
bryanrasmussen wrote: post #159

Earlier quoted context omitted.

if you're not being facetious, in English vernacular being a frustrated X (as in a frustrated academic, frustrated poet, etc.) means that the person is actually not that, but have in some way been prevented from being the thing (frustrated) and have perhaps rancor in regards to the frustration and towards those who actually are the thing (this second feature being sometimes implied depending on who is doing the descr…

As an English speaker, my only interpretation was that you were an academic frustrated about academia.

https://www.ldoceonline.com/dictionary/a-frustrated-artist-a... - came up if I search for 'meaning of frustrated poet' which is the kind of thing where the phrasing most often comes up.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#173

Post by KirillPanov »

narrator wrote: post #140

I think there's a narrative being set up with Rokus Basslik and all that stuff that AI will do very bad stuff and it's nobody's fault. It's actually going to be somebody's fault, but the plausible deniability is too tempting of a tool to not build this narrative. Look at Rumble suing Google over their search rankings recently. Google just shirks their shoulders and says it's the algorithm and even they can't explain…

> Rokus Basslik

"the AI could punish a simulation of the person, which it would construct by deduction from first principles" -- https://rationalwiki.org/wiki/Roko%27s_basilisk

Yeah, not afraid of that.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#174

Post by LargoLasskhyfv »

bserge wrote: post #171
LargoLasskhyfv wrote: post #169

Earlier quoted context omitted.

Personality changes as described don't really matter IMO, because broken . Regardless if machine or man. Does not work as intended anymore, or at all.

Well then you'd need to define "normal" and that's not easy because everyone is different. It can even be dangerous. Easy example: some people (still) define having a different religion or being gay as "broken". People that go through depression, suicide thoughts/attempts and then recover are also different, because they gained new information.

I didn't mean that in a personal way, more like zoomed out, trying to see the whole system/species.

...because they gained new information. Tell me all about it. Had a near death/out-of-body experience 20 years ago. It's engrained/burned in so hard, that I guess even a lobotomy wouldn't erase it ;->

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#175

Post by js8 »

bryanrasmussen wrote: post #159
spuz wrote: post #155

Earlier quoted context omitted.

But you said in your first comment that you were an academic. "I confess I am a frustrated academic"

if you're not being facetious, in English vernacular being a frustrated X (as in a frustrated academic, frustrated poet, etc.) means that the person is actually not that, but have in some way been prevented from being the thing (frustrated) and have perhaps rancor in regards to the frustration and towards those who actually are the thing (this second feature being sometimes implied depending on who is doing the descr…

Thanks, I wasn't aware of the 2nd meaning. I also interpreted the "frustrated academic" as "an annoyed academic", rather than "an unsuccesfull academic".

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#176

Post by ansible »

LargoLasskhyfv wrote: post #144

Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever? As in: "So Long…

> I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those.

I'd say we're at a decent risk of something worse than your scenario happening.

(This is not original to me) I think it is useful to consider the non-human intelligence that we've already created: The Corporation.

They tend to act (especially as they get larger) in a sociopathic fashion, primarily because of their reward structure: making money. They inflict great harm upon the environment and populace because things like pollution are usually not realized on their balance sheets. If a corporation can quietly dump toxic waste out back, while raking in the profit, it will often do so.

Only the humans in the system act as a check on the corporation's activities, and often they too are blinded by the desire for money. Just bribe the regulator, and maybe the corporation can continue dumping for a while longer. Even if they eventually get caught, the smarter humans will already have taken their profits, and moved out of the situation. There is not nearly enough being done to claw back these gains from people and corporations inflicting harm on others and the environment.

If we incentivize AGI the same way we incentivize corporations, expect the same results, only faster.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#177

Post by LargoLasskhyfv »

ansible wrote: post #176
LargoLasskhyfv wrote: post #144

Trollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever? As in: "So Long…

> I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those. I'd say we're at a decent risk of something worse than your scenario happening. (This is not original to me) I think it is useful to consider the non-human intelligence that we've already created: The Corporation. They tend to act (especially as they get larger)…

Good point! I'm actually aware of that, but didn't want to touch it.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#178

Post by tsimionescu »

happythomist wrote: post #117
astrophysician wrote: post #114

Earlier quoted context omitted.

> I don't think you can say that life is built on computational processes unless you use a definition of "computation" that is so vague and all-encompassing that it becomes effectively meaningless. I mean, there's nothing physically stopping us from simulating a brain, right? It's a finite object with a finite amount of physical ingredients, and therefore with a finite amount of computing power we can simulate what i…

I think you're missing the central point, which is that computation is observer relative. Anything can be interpreted as a computational process. Searle: "Thus for example the wall behind my back is right now implementing the Wordstar program, because there is some pattern of molecule movements which is isomorphic with the formal structure of Wordstar. But if the wall is implementing Wordstar then if it is a big enou…

> I think you're missing the central point, which is that computation is observer relative. Anything can be interpreted as a computational process.

This is completely wrong. The opposite is true: computation is a mechanical process, it does not depend on an observer giving it meaning. It's true that the same mechanical process can be interpreted as different computations, but they will have the exact same computational properties (e.g. complexity), only the result will be interpreted differently.

In particular, it is extremely unlikely that the wall is implementing Word Star, because WordStar is a highly structured computation. The wall MIGHT be implementing some very simple additions, and essentially any process is implementing any one step computation.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#179

Post by im3w1l »

mejutoco wrote: post #165

Not to be unpolite or as a critique to the specific paper, but I am more worried about superstupidity not being able to be contained. I like to be exposed to these very theoretical questions. I am aware that progress depends often on explorations of apparently useless topics (like some pure mathematics). But when I read expressions like ' neo-fear of superintelligence' and contrast the topic with real-world immediate…

It's a huge issue that may well wipe out humanity, but no one knows when it may hit and it's hard to explain so it's understandable you find it hard to relate.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#180

Post by axilmar »

Proving that an algorithm will do no harm is exactly the same as proving that an algorithm terminates: in both cases, we are trying to know the output of the program without running it. We may prove some programs, but we can't have a generic algorithm that does so.

And thus, if we have a program that represents an AI, we can't prove that it is ok to use it or not.

However, the problem is not the AI itself, it's the humans that feed problems to it to solve.

As the paper says, one example is 'make the world a happy place'. This can lead an AI to destroy all humans. But the problem is actually the specification of the problem, which is insufficient.

Again, the problem is us humans, and not the AI.

The only problem with AI is when AI gets a physical body and needs to survive competing with the rest of us: if it is a super intelligence, it will thrive despite all the humans' effort to contain it.

Post reply on HN