Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's Researchers: Protecting Against AI’s Existential Threat

wsj.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#51

Post by staunch »

Zaak wrote: post #45
staunch wrote: post #24

Earlier quoted context omitted.

1. Is super goodness an inevitable result of super intelligence? 2. Is super intelligence an inevitable result of creating self-evolving software? My feeling is that the answer is yes to both, but I agree that these are not settled questions.

Have you heard the paperclip maximizer thought experiment? Here is a good video on the subject: https://www.youtube.com/watch?v=tcdVC4e6EV4

This is the risk that a narrow intelligent machine will go nuts. A super intelligence would know not to do stupid stuff, by definition.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#52

Post by eli_gottlieb »

staunch wrote: post #35
eli_gottlieb wrote: post #32

Earlier quoted context omitted.

>1. Is super goodness an inevitable result of super intelligence? No. Evidence: $PERSON_YOU_HATE is very intelligent. Human moral instincts/reasoning may be neuroscientifically simple , having some core mechanism that "unfolds" across sensorimotor datasets. This is very plausible, because we can already see core sensorimotor systems that include interoception, and a core affective system to move the sensorimotor syst…

> Evidence: $PERSON_YOU_HATE is very intelligent. In every case I can think of, I think the person's lack of intelligence is the reason I dislike them. That they do not understand my ethical problem with them is the problem. Or their calculus is off, which also seems to be a lack of intelligence from my POV. > ...they will not have humanoid motivations... That's probably a good thing. We should want them to be better…

>The important thing is that they're ethical

And, non-tautologically, where do you think the ethical knowledge and motivation come from?

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#53

Post by eli_gottlieb »

staunch wrote: post #47
jerf wrote: post #37

Earlier quoted context omitted.

"What kind of evidence is there against #1?" All intelligence produced to date has had nothing like "morality" from anything I've seen, not even the basic seeds of it we can see in simple animals in the wild. There's even been some hand-wringing articles about it about AI-based approaches locking minorities out of loans and such, for instance. Certainly the military has not reported any problems to date with their AI…

Chimps hunt down and kill monkeys for food. It's really hard for humans to watch but no chimp would ever even consider the ethics of this, even though they do have a level of ethics (e.g. fairness among themselves) We're more intelligent than chimps, so we understand that monkeys are pretty evolved and suffer a lot when you eat them alive. Chimps aren't kept up at night by the screams of their victims. A healthy huma…

>So what does that tell us about intelligence? That your understanding of a monkey's suffering leads you to more ethical behavior.

Rubbish. We don't empathize because we're intelligent. We empathize because we're empathetic. That's a specific mental capacity, separate from causal inference.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#54

Post by Houshalter »

staunch wrote: post #2

The real goal should be making a super AI that is worthy of being humanity's collective offspring. The bias in favor of our bags-of-mostly-water is provincial. All we really need is quality genetic engineering of our super AI baby, so we don't end up with a Frankenstein monster. Humanity's procreation of a superior intelligence is objectively a good thing. Humans do not sit on any plateau of goodness or intelligence.…

>As long as this super intelligence leads to super ethics and super consciousness

It doesn't. Humans happen to have "morality" because we evolved in groups. Groups of humans that cooperated and cared for each other better reproduced more. And so we evolved empathy.

But this is an entirely arbitrary feature. There's no law of the universe that says you must be moral. There's no reason an AI would have anything like our sense of empathy or caring for other beings.

AI works by predicting what actions are the most likely to lead to a goal. Better AIs are better at finding paths to a goal. But the goal itself is always arbitrary. If you made an AI to run a paperclip factory, it would eventually convert the entire mass of the Earth into paperclips.

As far as "consciousness", it's likely something similar is true of that. The experience of consciousness is an artifact of how our specific brain structures work. Is it really the most efficient algorithm for intelligence? I doubt it. It's quite possible we could get replaced by an AI that builds and accomplishes great things - and there will be no conscious being left in the universe to appreciate it. A Disneyland with no childern, as Bostrom calls it.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#55

Post by mannykannot »

staunch wrote: post #28
mannykannot wrote: post #25

Earlier quoted context omitted.

That would be for you to demonstrate, but even if we accept it to be true for the sake of a discussion, it is not relevant, because there is no reason to believe a non-human intelligence would choose to accept those ethical percepts that humans would prefer.

Unless humanity's level of ethics derive from our level of intelligence. In which case, our only complain would be that we're too stupid to understand its superior ethics. The same way a dog might not understand the ethics of a doctor injecting life-saving medicine into it. The doctor is acting super ethically but cannot explain it to the dog.

We are under no logical obligation to accept that the ethics of entities that can outsmart us are superior, even if your premise is correct - and as that premise itself is merely speculative at this point, it would be wise to be prudent.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#56

Post by iza »

It's funny there is all this fear of an automated system devoid of morality, human values, and oversight taking over the world. It has already happened and its name is capitalism. The market decides who fails and prospers, who lives and dies. Humanity has handed over the reins a long time ago and it is only a matter of time before we are driven off a cliff.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#57

Post by Zaak »

staunch wrote: post #51
Zaak wrote: post #45

Earlier quoted context omitted.

Have you heard the paperclip maximizer thought experiment? Here is a good video on the subject: https://www.youtube.com/watch?v=tcdVC4e6EV4

This is the risk that a narrow intelligent machine will go nuts. A super intelligence would know not to do stupid stuff, by definition.

The problem isn't about doing stupid stuff. The problem is about doing very smart stuff when your core values massively conflict with humanity's continued existence.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#58

Post by cjbprime »

sjg007 wrote: post #39

The more I study deep learning the less I’m worried about emergent general intelligence. I do see advances in robotics and training AIs to learn repetitive tasks that are too difficult to write in code. I also see killer autonomous drones but I don’t see reasoning systems as yet being effective. You might call this generalization or polymorphic behaviors where the system can operate over new types it’s never seen bef…

You're commenting on a blog post about a Go program that taught itself everything about the entire problem domain from scratch, and then improved to superhuman (and even supercomputer) ability through self-play.

...

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#59

Post by It_ »

damodei wrote: post #33
It_ wrote: post #5

Too bad OpenAI has less than a handful of people working on safety and only one safety paper (to be published in a few months) throughout these two years of OpenAI's existence. Actions speak louder than words.

This is Dario Amodei, head of the OpenAI safety team. We are devoting a substantial fraction of the organization’s bandwidth to safety, both on the side of technical research (where we have several people currently working on it full-time, and are very actively hiring for more: https://openai.com/jobs/ ), and in terms of what we think the right policy actions are (several of us, including me, have been doing a lot of…

> substantial fraction of the organization’s bandwidth to safety

Although OpenAI's group ethos has a strong safety bent, there are only three research scientists working on technical safety research full-time, including yourself and a very recent hire. Before this summer, while you focused on policy and preventing arms races, there was only one person focusing solely on technical safety research full-time, despite the hundreds of millions donated for safety research. The team and effort should be larger.

> it’s important for our organization to be on the cutting edge of AI

I agree that OpenAI needs to be at the cutting edge, though always pushing the edge of AI to work on safety is needless when there is a significant backlog of research that can be done in ML (not just in RL). It's true capabilities and safety are intertwined goals, but, to use your analogy, the safety meter is not even a percent full. Topics outside of value learning using trendy Deep RL that OpenAI should pioneer or advance include data poisoning, adversarial and natural distortion robustness, calibration, anomaly and error detection, interpretability, and other topics that are ripe for attack but unearthed. There is no need to hasten AI development, and doing so does not represent the goals of the EAs or utilitarians who depend on you --- notwithstanding the approval of advising EAs with whom you have significant COIs.

OpenAI's safety strategy should be developed openly since, as of now, OpenAI has no open dialogue with even the EA community.

Re: OpenAI's Researchers: Protecting Against AI’s Existential Threat

#60

Post by sjg007 »

cjbprime wrote: post #58
sjg007 wrote: post #39

The more I study deep learning the less I’m worried about emergent general intelligence. I do see advances in robotics and training AIs to learn repetitive tasks that are too difficult to write in code. I also see killer autonomous drones but I don’t see reasoning systems as yet being effective. You might call this generalization or polymorphic behaviors where the system can operate over new types it’s never seen bef…

You're commenting on a blog post about a Go program that taught itself everything about the entire problem domain from scratch, and then improved to superhuman (and even supercomputer) ability through self-play. ...

Umm no I'm not.
Post reply on HN