OpenAI’s accidental attack against Hugging Face is science fiction that happened

Post by 8note » Fri, Jul 24, 2026, 5:33 PM UTC

manux wrote: post #246

simonw wrote: post #236

Earlier quoted context omitted.

I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off. OpenAI said: > We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. It was a test of raw capabilities of the underlying model.

I guess we could debate what counts as alignment, but I think my initial point remains that if the underlying base model needs these classifier guardrails so badly then the way we train the base models is creating fundamentally misaligned models that are happy to pursue illegal behavior. I'm sure OpenAI would argue that base model + guardrail is aligned, but considering the "relative intelligence" of these two pieces…

Post by mnicky » Fri, Jul 24, 2026, 8:56 PM UTC

s3p wrote: post #434

mnicky wrote: post #86

I think points that deserve more attention in the current public discourse are: - This should be a huge wakeup call for everybody. - We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. - It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox…

> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. Help me understand how a text-based model could somehow physically construct a string of nucleotides?

Post by reasonableklout » Sat, Jul 25, 2026, 8:55 AM UTC

docjay wrote: post #439

reasonableklout wrote: post #375

Earlier quoted context omitted.

Do you work at a lab? Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.…

I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes,…

Post by theptip » Sat, Jul 25, 2026, 5:08 PM UTC

lebovic wrote: post #388

theptip wrote: post #379

Earlier quoted context omitted.

I dunno man, the original announcement said things like: > To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy And > In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face…

Yes, I think you could probably get something similar from Opus 4.5 (2025). Definitely Opus 4.6. I still think recent models are more capable, though! Some of the model behaviors that make it better at pentesting, like persistence, can be improved with harness-level tricks (e.g. alloys, automated nudges, pre-fill to promote persistence, coordinated swarms, etc). You mentioned the UK AISI's evals. Their harness is lik…

Post by docjay » Sat, Jul 25, 2026, 7:17 PM UTC

reasonableklout wrote: post #465

docjay wrote: post #439

Earlier quoted context omitted.

I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes,…

Are you referring to table 1 from the ExploitGym site [1] for the 102.1 average mins for Mythos and 69.8 for GPT-5.5? These are under a two-hour time limit. In Figure 5, the authors experiment with extending the time limit to 6 hours and show that Mythos keeps improving. It seems pretty straightforward to me that OAI decided to run a variation of the eval with an even higher time limit. Regarding seeing the metrics t…