Timeline of the OpenAI accidental attack against Hugging Face

Yeah but persistence is immeasurable. They need to know when they’re hacking. Or better yet make the model providers liable - they’ll find a solution right quick

I think one of the most interesting details here might be tucked away in that first bulletin point:

> May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.)

The more I think about this the more I suspect that the fact this happened while training a new model is key to understanding what went wrong.

In RLVR - Reinforcement Learning with Verifiable Rewards - you set the model a goal and have it take any steps necessary to achieve that goal.

Clearly one aspect of OpenAI's training here is to RLVR their models for cybersecurity tasks. Just like pre-training benefits from dumping in vast sources of knowledge, the more tasks you can feed into RLVR the more of a general purpose capable model you get at the end.

This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process.

AND it explains (but does not excuse) why monitoring was so lax. If you're training a new model like this you presumably set it thousands of tasks like this in parallel. I can see how you might miss that a tiny subset of your training agents have started leaving each other messages in filenames on your packaging server.

Someone once told me that you can't just leave the racist materials out of your training data if you want a non-racist model: it has to have seen examples of racism in order to later be taught that racism is bad.

I can see echoes of that here. If your model doesn't know how to aggressively hack things how do you later teach it not to?

(I have little knowledge of how RLVR works in practice so I'm looking forward to hearing from people who can help me understand if I'm on the right track here.)

I think we are in need of Europe's leadership in safety legislation. It is foolish to say 'China will get ahead' when they will harm themselves too. Being unsafe is not something to gloat about.

Stiff fines for such incidents to pressure companies to get their acts together is a good start.

> And really good at marketing those failures into spin at how amazing their products are.

The paranoid style in American PR (with apologies to Richard Hofstadter)

The fact that the world has become susceptible to what amounts to a mob shakedown - look at how dangerous our amazing products are, don't you need them to protect you from others misusing our products? - is to me a really compelling example of US gun lobby thinking leaking out into a global problem.

Anthropic and OpenAI may be able to bounce this into restrictions on open weights models, but they are going to have a lot less luck extending this into foreign policy. If the USA can't control its weapons, they aren't going to see a lot of co-operation from foreign countries on a blockade of open weights modeld from China.

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons.

And we know that Chinese models are derived from OpenAI and Anthropic, they are at the same time talking about how dangerous models can be (even their aligned ones it seems), while being also responsible for the development of the whole industry and providing the basis for adversary countries to build their own.

I don’t believe we would accept that for any other technology that is expected to be as risky for the world