METR and Redwood Offer Holy %^ Postmortem of the HuggingFace Hack

"Being outmatched" is not a novel situation for humans either individually or collectively and there are a hell of a lot of ways we can approach that situation productively. OpenAI doesn't appear to have bothered.

Here's a freebie: if you're building something that might turn out to be Skynet and you don't know what it's capable of, your testing regime should assume it is capable of doing bad and unexpected things and account for that possibility: airgap if you can, monitor all network traffic, monitor all hardware usage statistics, log everything, constantly analyze logs, collect baselines and snapshots, also don't trust anything from a device that a model is running on without cross-correlating with other information as much as possible (does your AI inference server claim low utilization? put a temperature probe on it and see if it's staying cool or getting hot, maybe Skynet-Alpha is overwriting /proc to mislead you for reasons you don't yet understand!)

In other words, if you WANT to be able to nip things in the bud - buy some nippers and watch for buds. Whatever else this situation is, or may turn out to be, it is not a situation where OpenAI was on their guard and still got surprised.

So I'm really not sure how much of it can be believed, especially since AI agents are strongly biased about the capabilities of AI agents.

For a training run, you will often do this. You'll randomly sample some of the forward pass. You can also imagine finger printing the logs and labeling with attempt types. If a new attempt type is hitting a brick wall or solving super quickly, I would imagine you would sample 1-10 of them and read the traces.

> 2. Only a small fraction of AI agents was engaged in this attack. Most never found the secret message board - let alone coordinated there. So reviewing random agents would take a while to surface this.

You could also have a classification of what counts as "cheating" (coordination, accessing the internet, etc) and score the results. If you are seeing a spike in this (even in a small group of the evals) you could manually look at those. Or you could stop inference on cheating sessions.

> 3. "Output tokens" of AI agents have weird shit in them all the time. Telling "normal AI weirdness" from "concerning AI weirdness" is not a trivial task.

If you start seeing "now I have access to the internet" or something similar, maybe that's a good signal something is going wrong?

Also, obviously, it would not be outside of OpenAI's ability to just completely air gap this training system. For example:

1. No network connection.

2. GPS based NTP for time sync for your servers.

3. Mirror of all apt, pypi, go, c++, Rust, Java, etc packages. (4. Take your training data and use that for a mirror of the web. (http://example.com -> mirror server -> local training copy).

They had systems connected to the internet connected to this system which was not air gapped. Designing an air gap system would be super easy, well within the means of openai, and betrays the assumption that they think they are actually building something dangerous.