The idea that
something with greater cognitive capabilities than us might be dangerous to us occurs to many people: sci-fi writers in large numbers to be sure, but also Alan Turing and a large fraction of currently-living senior AI researchers.
What really gets me concerned is the quality of the writing on the subject of how can we design an AI so that it will not want to hurt us (just as we design bridges so that we know from first principles they won't fall down). Most leaders of AI labs have by now written about the topic, but the writings are shockingly bad: everyone has some explanation as to why the AI will turn out to be safe, but there are dozens of orthogonal explanations, some very simplistic, none of which I want to bet my life on or the lives of my younger relatives.
Those who do write well about the topic, particularly Eliezer Yudkowsky and Nate Soares of the Machine Intelligence Research Institute, say that it is probably not currently within the capabilities of any living human or group of humans to design an AI to be safe (to humans) the way we design bridges to be safe, and that our best hope is the hope that over the next centuries humankind will become cognitively capable enough to do and that in the meantime people stop trying to create AIs that might turn out to be dangerously capable -- which (because outside of actually doing the training run, we have no way of predicting the effects on capability of the next architectural improvement or the next increase in computing resources devoted to training) basically means stopping all AI research now worldwide and for good measure stopping progress in GPU technology.
Eliezer has been full-time employed for over 20 years to work on the issue (and Nate has been for about 15 years) and they've had enough funding to employ at least a dozen researchers and researcher-apprentices over that time to bounce ideas off of in the office.