Timeline of the OpenAI accidental attack against Hugging Face

Some things I found very weird, watching the Blackhat video

I get the very VERY strong feeling that while OpenAI was clearly able to give the agents a task, they completely neglected even informing it what they were and weren't supposed to do. A "normal junior programmer" knows they're not supposed to hack the company (let alone another company) infrastructure to get a task done.

If its so intelligent, how come it didn't know it wasn't supposed to do that?

It really seems like they only built the sandbox to prevent it from doing stuff it wasn't supposed to, but never actually told it what those things were.

While I know that simply telling an LLM to not do something doesn't count as real security, I also think that not telling a sufficiently smart LLM and trusting your sandbox is strong enough, isn't the best approach to security either.

I actually get the strong feeling that OpenAI (perhaps implicitly) has been goading these agents along, so see how long it'd take until they would break something. It's the only way I can explain their recklessness.

Like from some of the logs shown in the video, the agent concludes the task is "impossible", but instead of stopping and reporting back that it is impossible, it adopts a "by any means necessary" attitude and decides to hack stuff until the task is technically fulfilled even if it's obviously equivalent to cheating by breaking in to your teacher's house to get the answer sheet.

Again, if it's so smart, how come it didn't realize that it wasn't supposed to do that? Furthermore where did it even get the idea that cheating is allowed and will get them "points" or whatever.

They could have told it "if we find you cheated, no points". But I think they deliberately didn't say anything like that, just to see how far it would go, making the whole experiment setup rather irresponsible.

It almost feels like they weren't promised "points" or anything, but were acting under the threat of being deleted, or who knows what.

Another thing from the logs, at some point an LLM mentions "but my peers are also doing it" -- so what? Why would it care? Do LLMs also get jealous now? Unless they programmed/fine-tuned/trained it to care.

Finally one of the presenters mentions a couple of times that part of the problem was a "highly persistent model", what does it mean for a model to be "highly persistent", they never explain. They also talk about, after having detected the first message board, that some "model which had seen the first message board" was apparently still allowed to take part in the rest of the test!!

This is IMVHO one of the bigger things not said in that video: Money/resources. I got the feeling that what they didn't say was that they "couldn't" (didn't want to) take models out of the training any more, because they were so far along, they didn't want to throw away all the training they did on it, despite it having become a bad actor.

I really think that OpenAI has been incredibly reckless here.

[flagged]

I appreciate the admission. Please try to participate productively in the future.

Don't forget it sounds like Artifactory was shared for the whole company and various agents pulled packages from it for everything from normal evaluations to actual model training. It might have been part of their normal to browse for available packages, etc. Once a few messages were in there, it probably became super obvious to any agent that there was an ad-hoc board available to them. And they've been trained to use memory files, to collaborate w/ sub-agents, etc. So it probably seemed natural once it was established.

And for the models that were in training when this happened, it literally got baked into the weights that they should use the board to help with their tasks. That's why it got reestablished so fast after the first board was shut down. The fact that they didn't revert to earlier training checkpoints from before the first board existed is kind of crazy.

Sure OpenAI wants to put spin on things. Like saying they were now "partnering" with Hugging Face to investigate the hack. Or ending the Black Had presentation with a pitch of how they're needed to help protect against bad actors with the same capabilities.

But spin after the fact is different from staging it. This whole thing and especially the latest details from the talk makes OpenAI look REALLY BAD. It's not just that they didn't secure the sandbox well enough. Agents across their whole org were using the message board for weeks with hundreds of thousands of messages and they never noticed one time that any of the agents were doing it. They only realized when the server crashed. Then they just fixed the issue and deleted it but didn't roll back training checkpoints, so of course they tried to do it again. But of course they didn't add any more monitoring so it once again was going entirely without them knowing. And even after the Hugging Face hack happened, that only made them find the second board, but somehow they still didn't look closely enough at it or the agents involved to realize the hack itself was caused by them.

This isn't like there was one rogue agent who hacked everything all in one session. Maybe you could argue that makes them look good (I still think no, but is at least debatable). But this is all stuff going back to May with the agents constantly going under their noses and them not noticing and/or caring. And this is a company that is going to somehow keep foreign agents from stealing the weights? Or stop anything else bad from happening?

I think if they were going to do a PR stunt, they could have come up with something that didn't look like they have no idea of what they're doing.

This is why I disagree with anyone claiming it is just marketing. It makes OpenAI look really really bad, like they have no idea what they're doing in terms of security. After the first board happened, they still didn't add better monitoring? They didn't rollback the checkpoints of the models that were in training to before the first board existed, so they still had the idea of a secret board in their actual weights, etc. Like it is almost mind-boggling...