>There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.Of course it is a dishonest marketing trick, OpenAI can't function any other way, but for another reason. Forget about effectiveness! This capability is known since Mythos and was completely believable before it. The trick is in propagandizing self-sufficient malicious rogue AI as the only possible course of events, and the necessity of banning everyone except themselves from AI R&D. It did NOT decide to hack HF spontaneously on its way to make more paperclips, people were guiding it specifically to hack computer systems so it's been in that particular mode. I'm amazed I even need to point out this fact and the giant conflict of interests they have.
They like to compare it with an atomic bomb, which is another trick. The secrecy around the atomic bomb had a massive pushback by the brightest minds who designed it, in the US, USSR (where it was much harder), and also any other countries that had to do this. And that is in the post-war setting, with Hiroshima and Nagasaki being direct examples of what even a small and primitive bomb can do. Declassification was the only thing that allowed to develop industrial-scale nuclear power and do a huge number of other innovations.
I advise taking a pause to cool down from all the AI hype and read Restricted Data: The History of Nuclear Secrecy in the United States by Alex Wellerstein, to understand the atmosphere in the US scientific community at the time (it was similar in the USSR, which industrial history I studied, although less rigorously than Alex), and reflect on how it compares with those people in modern AI labs and even outside them, in the self-appointed AI safety community.