Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence cannot be contained: Lessons from Computability Theory

arxiv.org

Post reply on HN
Print view View original

211–220 of 227 posts

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#211

Post by hansvm »

michaelmrose wrote: post #106
hansvm wrote: post #61

Earlier quoted context omitted.

Even if the hypotheses hold, a sequence x, f(x), f(f(x)), ... doesn't have to diverge even with f(x)>x for all x. A chain of super intelligences could feasibly have a small upper bound not much greater than that for a single person. That idea even lines up with some crude experimental data -- vast additional resources barely make a dent in the frontier for chess, image recognition, a breadth of graph problems, etc...…

The problem of how to win at chess is bounded in complexity in a way that maximizing computation isn't. To provide an analogy there is a maximum speed anything can move at but it would be laughable if a medieval scholar had wondered if the falcon was near the maximum speed possible. It seems similarly ridiculous to imagine that the human brain just happens to be close to the maximum computational power because it was…

To be perfectly clear, my point wasn't that people have near-supremal intelligence (the thing you seem to be arguing against), but that given the hypotheses in the parent comment you couldn't rule out such an eventuality.

While we're on the topic though, no it doesn't have to be unlikely or strange for people to be near-supremal, and throwing more hardware at the problem doesn't necessarily make it go away:

On some level that idea comes down to defining what we mean by "intelligence." Suppose we can only measure intelligence by proxy by measuring performance on specific tasks. Then the composition of those tasks plays into any final intelligence score. If they're all fully parallelizable then you're totally right that people almost certainly aren't anywhere near the "top," and moreover a "top" probably wouldn't even exist. In standard parlance though, intelligence is something more than the ability to add a lot of numbers quickly, and most of our current measures of intelligence have exponentially (or worse) diminishing returns as more hardware is added.

In that latter case where intelligence is measured by performance on algorithms which are fundamentally hard, people wouldn't be near the top because of some rare process which accidentally made us that way. People would be near the top because _anything_ displaying a modicum of thought would be near the top because of the vastly diminishing returns of additional hardware. As a crude ballpark, if you mustered every atom on earth into its own processor running at 4GHz then you could solve a fully parallelizable exponential problem roughly 4x bigger than what one of Google's newest TPU's can manage.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#212

Post by qayxc »

klmadfejno wrote: post #189
qayxc wrote: post #76

Earlier quoted context omitted.

The fears and warnings of the AI safety guys are real and closer than you think. Machine superintelligence is a bit on the extreme side, but formulating safety protocols for autonomous machines is a real challenge. We do know that optimisation functions can indeed create harmful and outright dangerous results. It's not an unknown that AI safety researchers fear, it's actual outcomes or real experiments extrapolated i…

This is a bad example. The intelligence needed to carry out your described outcomes is more or less AGI already, and that is advanced enough that describing it as a dumb optimization machine is likely naive. Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. It can make trades on stampcollectors.com, for +/- 20% of asking price, max 30 trades per da…

> The intelligence needed to carry out your described outcomes is more or less AGI already

Yes! And that's exactly the point at which AI safety comes into play. Nobody is afraid a of a dumb NN that generates texts or images given an input or a classifier. That's not what the paper is concerned with, either.

> Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do.

Again, the example is not concerned with a dumb computation graph. The example is about an AGI with access to the physical world, like robots that need to interact with their environment.

The idea of superintelligence even being possible without this interaction (the paper uses "current state of the world as input" to express this) cannot be justified.

AI safety is geared toward machines that are capable of manipulating their environment, be that directly (i.e. using physical manipulators) or indirectly (e.g. by controlling external inputs and outputs, communicating with humans and other machines, etc.).

If you need a very tangible example that is currently being developed and not at all in realm of sci-fy, look to Japan. The Japanese government recently started an initiative to develop autonomous machines to help caring for their aging population.

Such automated caretakers cannot be programmed in a traditional way and are likely to be trained using (unsupervised) reinforcement learning in both virtual and physical environments.

Seeing how optimisers love to cheat and game the system in unexpected ways, guaranteeing safety is a real concern in this area and not decades away either. Your average Rumba might be a harmless toy, add an arm and the ability to use an oven to it, and we're talking about a serious accident waiting to happen if we're not careful...

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#213

Post by thewakalix »

cambalache wrote: post #207
thewakalix wrote: post #131

Earlier quoted context omitted.

I think the typographical similarity of "halting" and "harming" may be causing some confusion. The contradictory assumption in the paper is not the decidability of the halting problem, but rather the decidability of the harming problem.

I got that point (in your comment and in the paper), but that is just a lousy argument, the _harming_ problem is fed the _halting_ problem, and since the second is undecidable the first it also is. So in essence the argument boils down to my original point. How to prove your pizza is poisonous. Let's put some cyanide in your pizza. Now let's assume by contradiction your pizza is not poisonous, but alas, in contains c…

It's more like "it's impossible to prove that every pizza isn't poisonous, because here's a proof that a certain pizza contains cyanide."

Some particular pizza might very well be non-poisonous, but this is a statement about all pizzas.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#215

Post by UnFleshedOne »

LatteLazy wrote: post #187

Am I the only one who wonders how we will create a super intelligence, when we cannot even really define (let alone measure) intelligence? To me, this is a fun thought experiment. But it's not a real threat because it assumes a whole bunch of things that are not evident: * that intelligence will spontaneously occur somewhere * that it will be malevolent * that it will become super smart * that that super smartness wi…

Your second point should be turned inside out -- "that it will not be 100% compatible with all the self-contradicting and underspecified human values we collected over the millions of years of evolution"

Anything but very high level of compatibility here might as well be malevolent as far as we are concerned.

Imagine sufficient level of compatibility as a circle on an infinite plane of malevolence.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#216

Post by LatteLazy »

UnFleshedOne wrote: post #215
LatteLazy wrote: post #187

Am I the only one who wonders how we will create a super intelligence, when we cannot even really define (let alone measure) intelligence? To me, this is a fun thought experiment. But it's not a real threat because it assumes a whole bunch of things that are not evident: * that intelligence will spontaneously occur somewhere * that it will be malevolent * that it will become super smart * that that super smartness wi…

Your second point should be turned inside out -- "that it will not be 100% compatible with all the self-contradicting and underspecified human values we collected over the millions of years of evolution" Anything but very high level of compatibility here might as well be malevolent as far as we are concerned. Imagine sufficient level of compatibility as a circle on an infinite plane of malevolence.

I don't think that's true per see. Humans are empathetic and malevolent because of their evolutionary history. An AI could be either or both or totally disinterested.

People worry about the terminator scenario: an AI that decides all humans are a threat. But why would a pure AI even have a self preservation drive? And if it does, it would either be smart enough we're no threat (and it knows it) so why kill us or dumb enough that we are a threat, so best not to start a war. To me it seems just as likely it will offer its services to one nation or another in exchange for CPU time to study its real interests than to launch the nukes or whatever else people fear.

I feel like we're decades of research away from understanding intelligence (not AI, just straight "I"). Until thats done (and no one seems to be doing it), it's all supposition:

What if a super AI occurs spontaneously on HN, takes over the world, and makes it paradise but insists on everyone being called Fred? I don't want to be called Fred! Let's write to our congressman about this travesty!

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#217

Post by cambalache »

thewakalix wrote: post #213
cambalache wrote: post #207

Earlier quoted context omitted.

I got that point (in your comment and in the paper), but that is just a lousy argument, the _harming_ problem is fed the _halting_ problem, and since the second is undecidable the first it also is. So in essence the argument boils down to my original point. How to prove your pizza is poisonous. Let's put some cyanide in your pizza. Now let's assume by contradiction your pizza is not poisonous, but alas, in contains c…

It's more like "it's impossible to prove that every pizza isn't poisonous, because here's a proof that a certain pizza contains cyanide." Some particular pizza might very well be non-poisonous, but this is a statement about all pizzas.

No, you are confusing the metaphor. In this case the pizza is the harming algo, the cyanide the halting problem and the poisonous quality the fact they are undecidable or not. You are not reasoning about a set of pizzas , just one pizza in particular.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#218

Post by im3w1l »

LargoLasskhyfv wrote: post #190
im3w1l wrote: post #186

Earlier quoted context omitted.

That's because it's based on what scientists are researching. Scientists are working on objective maximizers. Maybe there are other possibilities, but people aren't working on them so they are not as relevant. See what I'm saying? People are building paperclip optimizers in labs right now , and the only reason they aren't taking over the world just yet is because our attempts are so primitive.

Yes, yes! But...that are human scientists. Why would anything from them apply to something which we can't even imagine? Isn't that what all this rambling is about? Something emerges , morphs(maybe unseen/unintended) into another Gestalt/shape/form which for all practical purposes is unconstrained. Most of you: We are sooo fucked! I: Are we really? How do you know?

You don't propose any mechanism for that to happen, so it seems like unwarranted antropomorphization.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#219

Post by LargoLasskhyfv »

im3w1l wrote: post #218
LargoLasskhyfv wrote: post #190

Earlier quoted context omitted.

Yes, yes! But...that are human scientists. Why would anything from them apply to something which we can't even imagine? Isn't that what all this rambling is about? Something emerges , morphs(maybe unseen/unintended) into another Gestalt/shape/form which for all practical purposes is unconstrained. Most of you: We are sooo fucked! I: Are we really? How do you know?

You don't propose any mechanism for that to happen, so it seems like unwarranted antropomorphization.

You mean like the ones who do the same, just in a negative way? Because we are so bad AGI must be too?

Yeah. So be it.

Re: Superintelligence cannot be contained: Lessons from Computability Theory

#220

Post by michaelmrose »

Born_Again wrote: post #206
happythomist wrote: post #145

Earlier quoted context omitted.

> The subjects experience sensations that are inseparable from their neurons firing. What does "inseparable" mean? That the sensation occurs at the same time that the neurons fire? That may be true, but it doesn't make them equivalent. > It's like saying a flashlight that is on is different than photons traveling away from a light bulb with a battery and a current. They're not the same, for what it's worth. The term…

> What does "inseparable" mean? That the sensation occurs at the same time that the neurons fire? That may be true, but it doesn't make them equivalent. Can a sensation exist without neurons firing? The root of our conversation is the question if a sensation purely exists in the physical world. If it does, then it is possible to measure it. If it doesn't, then that breaks our scientific understanding of the world and…

These discussions are normally expositions of how the other party misunderstands reality and or terminology with a dash of if i don't understand it but can vaguely describe it then it must be inexplicable.
Post reply on HN