Thoughts on OpenAI, reinforcement learning, and killer robots

Thanks for the answer.

Most humans do not really come up with our goals on our own either. We are heavily influenced by genes and environment. People's personality are partially changed by the environment and occupation they are in. Twins adoption studies also show that genes have surprisingly strong effects on personality even after decades living apart.

By your definition, humans do not possess general intelligence.

Edit-Reply:

I don't think we can resolve an age-old philosophical question on Free Will in a comment thread, so I'll leave it at that. :) However, General Intelligence and Free Will are largely orthogonal.

We can imagine an alien who is highly capable of any tasks human can perform and beyond, including coming up with creative subgoals as necessary, but will only serve their master's ultimate goal(s). An AGI could be like that alien but built to serve human masters. The twist is that when someone orders an AGI to efficiently reduce animal suffering, without value alignment with humans, it could come up with euthanasia as a subgoal to reduce the suffering of starving kittens.

> Wouldn't you even agree that there is a 1% chance that AGI is possible and that it could harm us once invented?

There's a small chance aliens could find us, too. Yet, we don't know any more about that than we do about AI.

The most imminent threat to humans is humans. In the nearer future, it seems more likely that someone will use machine learning technology in weaponry to cause harm than create AGI. Or, that humans will undergo a global "cultural revolution" like in China or Cambodia (or ISIS now), killing anyone with an education, under the guise that "machines are evil", yet with the true goal of obtaining power, as humans often do.

Ultimately, people are free to work on what they want. If some people want to work on AGI safety, that's great. Perhaps their goal is to make humanity a little more kind. That's certainly worthy.

For me, I'm more interested in seeing research focused on giving better tools to doctors for diagnoses. Radiology is a great place to apply image recognition systems. We just need some good labeled datasets.

Sometimes I get frustrated by the "AGI is an existential risk" quotes like Musk's because I feel that's a distraction from some really beneficial applications of machine learning that will help save lives. It could cause over-hype, leading to another AI winter, pushing back life-saving advances 15 years, as happened after the 80s.

Personally I don't have a problem with the kinds of things some AGI safety researchers, like Stuart Russel, says about AGI. He isn't saying it will arrive in 2030-2040, like Musk.

Musk has an incentive to capitalize on people's fear of AGI. It attracts investment in his non-profit, which can in turn help him continue building his in-house AI program at Tesla. Primarily I disagree with his conclusions about AI tech, but I'm also weary of his motivations. I think some people view him as altruistic and I just don't see that.

> What would be the minimum level sufficient to convince you that AGI is possible?

A system that can determine its own goals. We don't have anything close to that yet, and I can't see anything short of that being a good indicator that we are close to AGI.

What about using machine learning to estimate time to solve difficult task with ai?

The Myth of Superhuman AI has to be a joke. Please tell me it's a joke.

> Intelligence is not a single dimension, so “smarter than humans” is a meaningless concept.

So "I'm smarter than my dog" is a meaningless sentence? My dog will be happy to hear that . . . if he could understand it.

> Humans do not have general purpose minds, and neither will AIs.

This statement (even if true) seems totally irrelevant to his argument. So I was really looking forward to read the long form version in section #2 and see if it somehow makes sense.

It doesn't seem to.

His point is that minds involve tradeoffs, so specialized agents might be better than a big general AI. Whose side is he arguing on here? He isn't showing that a general intelligence couldn't exist that's better than humans in every way. He's just saying that even it would be inferior in specialized tasks to even more inhuman, specialized minds. Comforting this is not.

> Emulation of human thinking in other media will be constrained by cost.

This section is kind of a hack too, but since at least the title makes sense lets give him a freebie.

> Dimensions of intelligence are not infinite.

Oh golly. He starts off by asserting that the core of the AGI people's ideas relies on intelligence being on an unlimited scale. It includes this gem: "Temperature is not infinite — there is finite cold and finite heat." My estimate of this being a joke article is going back up.

Alright, I agree that intelligence, like temperature, is not infinite. I still don't want be be competing for a job opening with Von Neumann. Von Neumann doesn't want to be competing for a job with future-AGI-silicon-Von-Neumann, whose brain is 500 million GPUs in a datacenter stretching across Montana.

> Intelligences are only one factor in progress.

Right, just wildly the most important one.

My honest opinion is that I feel a little bad for the author. He clearly just doesn't want to believe the plausibility of this idea, and is just throwing every argument he can against it and hoping one of them sticks. I sympathize, but this kind of squid-ink writing probably doesn't serve as a great foundation for future debate.

Unfortunately, no one can really articulate the full set of human values yet, especially since they are often in conflict in real-world situations. Which value(s) should be prioritized over others? How about differing values between cultures and human groups within each culture?

You might have heard of the Trolley Problem. There are many and more complicated variations of that.

This Harvard session on Justice (The Moral Side of Murder) is edifying and surprisingly fun to watch: https://www.youtube.com/watch?v=kBdfcR-8hEY

The problem with AGI is more acute than most other technologies because it is software-based, which makes it nigh impossible to regulate effectively, especially globally.