There's No Fire Alarm for Artificial General Intelligence

You know this how? Where is the science behind it?

I think there's noone alive today who has any idea how we are going to go from where we are today, to AGI.

Recent advances are remarkable- but much more so if you're a specialist. The impact on most peoples' lives is much smaller (and it comes primarily from dumb automation, rather than actual intelligence).

Machine learning is not enough to get to AGI: relying on huge amounts of data and processing power is just not a sustainable way to keep increasing your capabilities.

All the success stories of the last 10 years are at least 20 years old (case in point: RNNs). Most of those successful techniques were found by sheer blind luck (famous example: backprop was not invented in the ANN community; it took twenty years for the idea to percolate all the way to them).

In the end nobody currently alive has a clue how we can get to AGI, or if that is even possible. Chances are, it will take many, many generations until we do- or a sudden, gigantic paradigm shift, of the kind that comes once every couple of centuries- think Newton or Einstein. Except AI is not physics. In statistical machine learning there is very little theory to guide the way, so people just try things hoping that something will work. And that's no way to make a quantum leap.

In this situation, to talk of the dangers of AGI is at least premature. Yes, it's not completely impossible that AGI will happen in our lifetimes. The same, however, can be said of an alien invasion. Should we start discussing setting up planetary defenses, when we haven't even found sign of alien life, yet?

By all means- let's have a conversation. There are people around whose job it is to have that sort of conversation. But let's all be aware (and let those people also be aware) that the conversation is most probably a couple hundred years early and by the time it becomes truly relevant, things will have advanced so much that it will just look pointless.

You may need exposure to different training data in order to go from mastering chemistry to mastering physics, but you don't need a fundamentally different brain design or approach to reasoning, any more than you need fundamentally different kinds of airplane to fly over one land mass versus another, or fundamentally different kinds of scissors to cut some kinds of hair versus other kinds. There's just a limit to how much specialization the world actually requires. And, e.g., natural selection tried to build humans to solve a much narrower range of tasks than we ended up being good at; so it appears that whatever generality humans possess over and above what we were selected for, must be an example of "the physical world just doesn't require that much specialized hardware/software in order for you to perform pretty well".

If all of that's true, then the first par-human biotech-innovating AI may initially lack competencies in other sciences, but it will probably be doing the right kind of thinking to acquire those competencies given relevant data. A lot of the safety risks surrounding 'AI that can do scientific innovation' come from the fact that:

- the reasoning techniques required are likely to work well in a lot of different domains; and

- we don't know how to limit the topics AI systems "want" to think about (as opposed to limiting what it can think about) even in principle.

E.g., if you can just build a system that's as good as a human at chemistry, but doesn't have the capacity to think about any other topics, and doesn't have the desire or capacity to develop new capacities, then that might be pretty safe if you exercise ordinary levels of caution. But in fact (for reasons I haven't really gone into here directly) I think that par-human chemistry reasoning by default is likely to come with some other capacities, like competence at software engineering and various forms of abstract reasoning (mathematics, long-term planning and strategy, game theory, etc.).

This constellation of competencies is the main thing I'm worried about re AI, particularly if developers don't have a good grasp on when and how their systems possess those competencies.