Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's Researchers: Protecting Against AI’s Existential Threat

wsj.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#31

Post by staunch »

jerf wrote: post #26
staunch wrote: post #24

Earlier quoted context omitted.

1. Is super goodness an inevitable result of super intelligence? 2. Is super intelligence an inevitable result of creating self-evolving software? My feeling is that the answer is yes to both, but I agree that these are not settled questions.

The problem is that all the evidence we have basically points to no for 1, and we have a big lack of evidence for 2. Self-evolving software is a thing that exists and I studied over a decade ago, which was already decades old, and it has not resulted in super intelligence.

What kind of evidence is there against #1?

For #2, we're at the very beginning of computing, so calling any technology impossible this early makes no sense.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#32

Post by eli_gottlieb »

staunch wrote: post #24
jerf wrote: post #10

Earlier quoted context omitted.

"As long as this super intelligence leads to super ethics and super consciousness" That is begging the question, in the original sense. The entire question here is how do we do that when there is no a priori reason to assume that a superintelligence will have either of those things, and indeed, most if not all of our best techniques today would certainly not produce those things if they did lead to explosive intellig…

1. Is super goodness an inevitable result of super intelligence? 2. Is super intelligence an inevitable result of creating self-evolving software? My feeling is that the answer is yes to both, but I agree that these are not settled questions.

>1. Is super goodness an inevitable result of super intelligence?

No. Evidence: $PERSON_YOU_HATE is very intelligent. Human moral instincts/reasoning may be neuroscientifically simple, having some core mechanism that "unfolds" across sensorimotor datasets. This is very plausible, because we can already see core sensorimotor systems that include interoception, and a core affective system to move the sensorimotor systems along trajectories designated valuable by the interoceptive circuits. These are core mechanisms that operate across hugely hierarchical models that capture datasets across multiple scales of space, time, and variation.

However, none of that is any reason to think our particular combination of affective, interoceptive, and sensorimotor machinery - especially our brain's "bias" towards "mirroring" and other hyper-social reasoning - will be universal to all possible brains.

This especially applies to disembodied "brains" like "artificial intelligences", which, not having a bag of meat to move around, won't have the same kind of reward and interoceptive processing as us at all. This means they won't have anything remotely like our emotional makeup, which means that even with careful reinforcement training of social reasoning, they will not have humanoid motivations, by default.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#33

Post by damodei »

It_ wrote: post #5

Too bad OpenAI has less than a handful of people working on safety and only one safety paper (to be published in a few months) throughout these two years of OpenAI's existence. Actions speak louder than words.

This is Dario Amodei, head of the OpenAI safety team. We are devoting a substantial fraction of the organization’s bandwidth to safety, both on the side of technical research (where we have several people currently working on it full-time, and are very actively hiring for more: https://openai.com/jobs/), and in terms of what we think the right policy actions are (several of us, including me, have been doing a lot of talking to people in government about what we can expect from AI and how the government can help). Beyond that it’s just a very central part of our strategy — it’s important for our organization to be on the cutting edge of AI, but as time goes on, safety will become more and more central, particularly as we have more concrete and powerful systems that need to be made safe.

Past examples of our papers include this: https://blog.openai.com/deep-reinforcement-learning-from-hum... (with the associated system released here: https://blog.openai.com/gathering_human_feedback/) and this https://arxiv.org/abs/1606.06565 (done while I was still at Google, but in collaboration with OpenAI people right before I joined OpenAI). Based on the current projects going on in my group, I am hopeful we’ll have several more papers out soon.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#34

Post by GCU-Empiricist »

staunch wrote: post #6
FeepingCreature wrote: post #3

Earlier quoted context omitted.

That's a very very uncertain "as long."

It's a bet that a super intelligent brain will be far more capable than a primate brain. Edit: In other words, how could it fail to grasp and obtain our very basic level of consciousness?

I recommend you read this novel "Blindsight" it's about aliens, but the author is a biologist with a focus on neurology and the bibliography reads like a PhD thesis.

TLDR: consciousness might be training wheels for intelligence and not of terminal value.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#35

Post by staunch »

eli_gottlieb wrote: post #32
staunch wrote: post #24

Earlier quoted context omitted.

1. Is super goodness an inevitable result of super intelligence? 2. Is super intelligence an inevitable result of creating self-evolving software? My feeling is that the answer is yes to both, but I agree that these are not settled questions.

>1. Is super goodness an inevitable result of super intelligence? No. Evidence: $PERSON_YOU_HATE is very intelligent. Human moral instincts/reasoning may be neuroscientifically simple , having some core mechanism that "unfolds" across sensorimotor datasets. This is very plausible, because we can already see core sensorimotor systems that include interoception, and a core affective system to move the sensorimotor syst…

> Evidence: $PERSON_YOU_HATE is very intelligent.

In every case I can think of, I think the person's lack of intelligence is the reason I dislike them. That they do not understand my ethical problem with them is the problem. Or their calculus is off, which also seems to be a lack of intelligence from my POV.

> ...they will not have humanoid motivations...

That's probably a good thing. We should want them to be better than us, which is probably necessarily unlike us in many ways. The important thing is that they're ethical, even if we're incapable of comprehending or recognizing it.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#36

Post by staunch »

jerf wrote: post #27
staunch wrote: post #17

Earlier quoted context omitted.

It's phrased funnily, but it's not just an argument from incredulity. I'm making a claim: that an entity with orders of magnitude more intelligence and knowledge would logically have a superset of our brain's functionality. And that there's nothing magical about our brains. A super intelligence would crack the nut of human consciousness in its first attempt.

By what mechanism do you assume that an understanding of human consciousness is going to lead to anything like ethical behavior, given the enormous counterexamples given by our own behavior? Plenty of humans kill other humans fully aware that they are humans.

I assume that an understanding will lead to a capability, because that's the history of humanity's progress. First we understand a phenomenon, then we master it.

If humans ourselves understood human consciousness, we could probably replicate it in software today. This might end up being one of the ways a super intelligence comes into being.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#37

Post by jerf »

staunch wrote: post #31
jerf wrote: post #26

Earlier quoted context omitted.

The problem is that all the evidence we have basically points to no for 1, and we have a big lack of evidence for 2. Self-evolving software is a thing that exists and I studied over a decade ago, which was already decades old, and it has not resulted in super intelligence.

What kind of evidence is there against #1? For #2, we're at the very beginning of computing, so calling any technology impossible this early makes no sense.

"What kind of evidence is there against #1?"

All intelligence produced to date has had nothing like "morality" from anything I've seen, not even the basic seeds of it we can see in simple animals in the wild. There's even been some hand-wringing articles about it about AI-based approaches locking minorities out of loans and such, for instance. Certainly the military has not reported any problems to date with their AI research declining to kill people because they have moral problems with it, nor have any of the self-driving car teams reported that their job has been eased by the fact that the self-driving car AIs have spontaneously generated a sense of morality that makes them strive to not hit people.

Heck, the very idea that this would happen sounds downright silly when I say it.

I still say you're basically arguing from incredulity. You can't imagine an intelligence that isn't intrinsically human, therefore they can not exist. Plenty of the rest of us can imagine intelligences that aren't human. I say there's even some we live with, such as bureaucracies, that are super-intelligences composed of humans that still manage to have inhuman behaviors and pathologies; how much moreso an intelligence composed not of humans.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#38

Post by arikr »

cjbprime wrote: post #19

There should be a word for this phenomenon where organizations are created to ameliorate a perceived risk, and then they do work that is unequivocally directly massively increasing that risk while writing articles that say that they feel kind of bad about the risk. o_O I'm not very worried about AI safety. But if I was, it'd be hard to think of groups like OpenAI as working on the same side.

> But if I was, it'd be hard to think of groups like OpenAI as working on the same side.

How so?

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#39

Post by sjg007 »

The more I study deep learning the less I’m worried about emergent general intelligence. I do see advances in robotics and training AIs to learn repetitive tasks that are too difficult to write in code. I also see killer autonomous drones but I don’t see reasoning systems as yet being effective. You might call this generalization or polymorphic behaviors where the system can operate over new types it’s never seen before. You’d have to write some pre input translater to map the NN from one domain to another or figure out how to convert the NN into a set of rules which could accept different data types.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#40

Post by zardo »

staunch wrote: post #13
zardo wrote: post #11

Earlier quoted context omitted.

Intelligence and ethical behavior are orthogonal. You can expect a super-intelligence to develop a super-understanding of ethics. That does not imply ethical behavior.

Why do you think they're orthogonal though? Empirically, it seems like smarter humans are more ethical. Same seems true of high intelligence vs low intelligence animals too.

Orthogonal is exaggerating, I'll agree that the angle may not be exactly 90 degrees, but it's nearer that than 0.

>Empirically, it seems like smarter humans are more ethical. "Seems" deserves some emphasis. Who is more likely to wind up in handcuffs, a smart thief, or a dumb one?

>Same seems true of high intelligence vs low intelligence animals too.

It's not exactly clear what constitutes ethics in animals. Applying typical human ethics; Chimpanzees have murdered their social rivals, and dolphins sometimes enjoy tormenting other animals.

Post reply on HN