Timeline of the OpenAI accidental attack against Hugging Face
simonwillison.net
Timeline of the OpenAI accidental attack against Hugging Face
1–10 of 293 posts
Re: Timeline of the OpenAI accidental attack against Hugging Face
#2Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
Re: Timeline of the OpenAI accidental attack against Hugging Face
#3Is it normal for these training/eval runs to go on for over a month?
Re: Timeline of the OpenAI accidental attack against Hugging Face
#4In a typical office environment, the correct response to “I don’t have access to this Google Doc” is to ask for access from the person who sent you the link. In another context, it could be fair to think “Hmm, this is some sort of capture the flag challenge, and obtaining access is the point of the assignment.” That assessment separates what we’d consider reasonable from way out of line.
I do wonder what this means for AI agents longer term. In a world where we humans already struggle with truth and misinformation, what happens when you can easily (intentionally or accidentally) spin up a cohort of fanatical believers to pursue any given conspiracy theory?
Re: Timeline of the OpenAI accidental attack against Hugging Face
#5so how many of these *Ellen Louise Ripley thinks about grabbing the flammenwerfer" events are we going to be getting over the coming months
Re: Timeline of the OpenAI accidental attack against Hugging Face
#6Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
Also shows how infrastructure collapses under its own weight. Reducing the number of moving parts would have helped. why a webdav endpoint is available from the vm anyway? and the fact that someone posted their credentials on pastebin and didn't rotate them after... put the agent in a linux namespace, allow one ip for whatever file sharing it needs, deep test that... then deploy
Re: Timeline of the OpenAI accidental attack against Hugging Face
#7This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended.
Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#8Is it normal for these training/eval runs to go on for over a month?
The way I read it was different things happening over several runs, such as the agents comparing notes so to speak, using artifactory
Re: Timeline of the OpenAI accidental attack against Hugging Face
#9Re: Timeline of the OpenAI accidental attack against Hugging Face
#10Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
Yes. It is very easy to add to the instructions "for every potential exploit you discover and use, document them as you go into this repository" and have alerting there. The fact that they did not do this means they wanted to be surprised, and have plausible deniability on their side when things inevitably blow up.
And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.