>Why on earth do we need an AGI to become "enlightened beings", has this gentlemen never read any literature on "enlightenment"?I think you might be misconstruing his definition of 'enlightenment'. In this instance I believe he is referring to humanity having the time and freedom for the pursuit of knowledge and intellectual reasoning. There are a number of instances in fictional literature were the authors discuss worlds in which civilization has achieved a state of enlightenment and dedicate their lives exclusively to science, philosophy and the arts (Olaf Stapleton's First and Last Men, and Starmaker provide several)
>but for what now? Just intellectual curiosity ?
The purpose is out of exercise to understand it. But again, if we might produce an AGI which is capable of digesting massive amounts of knowledge, but with the ability to reason we might be able to achieve certain goals like fusion energy sooner than expected. IMO solving the energy problem in itself is worth the risk as our uncontrolled use of fossil fuels is an existential threat to life on this planet, not to mention it would rebalance the distribution on wealth and power around the world.
As for Hinton, I think you are spot on with why he reacted this way. However, I facepalm every time the 'killer robots' example comes up because it is the worst example of the threats these things pose compared the following:
- AI being used to manipulate public opinion on a massive scale creating a new age of demagoguery.
- AI being used to fabricate different realities within an information sphere by generating text, imagery and audio/video media supporting a certain set of narratives. Essentially a type of super propaganda.
- AI being granted trust to by humans to perform certain functions which it is not capable of, because humans are too ignorant to subject it to proper scrutiny.
- AI being used to manipulate either markets, financial institutions or economies at a massive scale, dramatically shifting the balance of power around the world.
Essentially, the general public's concept of warfare needs to be broadened to consider non-kinetic conflict. Warfare is effectively a pursuit in the change of policy in other nations(or groups) by many different means. The last resort of these methods is inevitably a kinetic engagement, where all other approaches have failed to achieve the desired result.
Altering this viewpoint allows us to reassess what we consider to be weapons rather than tools, and gives us a better idea of where we should expect these threats to emerge first.