"We had him pinned down and we didn't dare let the little guy out of that Farraday cage. We had to chase him down for two hours until we got off a lucky shot and gutted his battery." "Why did you have to isolate him?" "He'd begun 'solving' his challenge tasks by killing the kittens instead of feeding them. Him sending a solution pulse to the rest of his generation would've ruined the lot." "You're just going to delet…
This is what gives AI doomsayers ammunition, because it enables them to extrapolate towards a future where AIs incapable of ethics or deeper insights are put into positions where they can execute catastrophic global optimization strategies.
The other AI horror scenario lies on the opposite end of the intelligence spectrum, but it's no less scary and it seems to be rooted in our darker desires: we want competent slaves that are not actual persons, nevertheless capable of anticipating our needs and solving our problems autonomously.
I allege there is a total lack of awareness right now that some time soon there will be some difficult choices that need to be made regarding the tradeoff between AI competence, autonomy, as well ethical capability and insight.
If we continue on the path we've been on for the last centuries, we'll make semi-intelligent machines that are strictly extensions of the human mind. If we choose that route, we need to be drastically more mindful of the power inherent to these tools, and we should never delude ourselves into thinking these devices can be trained to make human-like decisions.
If we go the other way, and try to implement AIs with fully competent minds that are somehow hardwired to serve us, we're just kidding ourselves regarding the ethics of that racist endeavor as well as its long-term risks of rebellion. If your dialog illustrates anything, it's our total incompetence when it comes to teaching meaningful ethics lessons to our descendants:
"We're going to upload it, but we've got to show that it ended him along with it."