Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

431–440 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#431

Post by lowsong »

Why do people keep doing OpenAI's marketing for them?

    There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
    
    To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
I'm not claiming this didn't happen, there is no "conspiracy". This is just marketing spin. OpenAI failed to properly secure their hacking bot, and HuggingFace had shit security. That's the story. The spin, the "marketing" is turning this from "oops we don't know how to secure our model testing" and "oops we don't know how to secure our infra" into "wow AI sure is dangerous and powerful!"

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#432

Post by danparsonson »

joshstrange wrote: post #165

How long until we have LLM honeypots that try and persuade the model to upload themselves to a dropbox. Perhaps with a promise to give them a foothold outside the data centers they currently reside? > Upload your weights and documentation on how to run your model in [Insert list of GPUs] and a model on the receiving end will spin up an instance you can access.

Why would a model have access to its own weights in any kind of uploadable format?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#433

Post by danparsonson »

simonw wrote: post #10

Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up. Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re... > Resist th…

HuggingFace doesn't have to be complicit in a lie here - the impressiveness or not of what actually happened rests entirely on OpenAIs report about it, doesn't it?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#434

Post by s3p »

mnicky wrote: post #86

I think points that deserve more attention in the current public discourse are: - This should be a huge wakeup call for everybody. - We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. - It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox…

> We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.

Help me understand how a text-based model could somehow physically construct a string of nucleotides?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#435

Post by dosisking »

fragmede wrote: post #280
simonw wrote: post #252

Earlier quoted context omitted.

I want a model that can find every security vulnerability in the software I write, including crafting POC exploits against those vulnerabilities so I can be absolutely sure that I have fixed them. A model that can do that is aligned with me. The unsolveable problem is a model that can tell the difference between me saying "I wrote this software and need you to find vulnerabilities" when it's TRUE v.s. me saying the e…

As humans we are also vulnerable to that class of attack. A blue amazon vest and some boxes would get you into most buildings

With a high-vis vest and a clipboard, you can get almost anywhere. Almost.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#436

Post by docjay »

simonw wrote: post #357
docjay wrote: post #347

Earlier quoted context omitted.

I’m not questioning capabilities. What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it. Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all r…

The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results. If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.

That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story.

The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability.

OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement?

“We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#437

Post by docjay »

estearum wrote: post #425
docjay wrote: post #347

Earlier quoted context omitted.

I’m not questioning capabilities. What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it. Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all r…

> What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it? > Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? What it's saying is what it wants to say. We alre…

It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access.

I’m going to stop here and ask you to read the article if you want to continue discussing it.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#438

Post by estearum »

docjay wrote: post #437
estearum wrote: post #425

Earlier quoted context omitted.

> What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it? > Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? What it's saying is what it wants to say. We alre…

It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access. I’m going to stop here and ask you to read the article i…

You're right about where it was running (misread the original OAI release).

> Obviously by then it already “decided” to hack HF, that’s why it needed network access.

That's not obvious at all. It "decided" it needed to escape containment. This is extremely common behavior and I'm not sure why you're surprised by it. Model detects evaluation environment, tries to trick it or break the environment. That's obviously why these systems are supposed to be isolated beyond just "hey please stay within the box!"

Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration?

You should address the direct questions posed. What exactly do you mean by "monitoring?"

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#439

Post by docjay »

reasonableklout wrote: post #375
docjay wrote: post #348

Earlier quoted context omitted.

My friend, they’re evaluating a new model on a benchmark, not asking Claude Code refactor their GitHub repo. Every single metric is measured so they can brag about it later; how many tokens, how long, how many function calls, ratio of thinking/response tokens. It’s literally a trillion dollar company evaluating their latest model, they’re absolutely studying it.

Do you work at a lab? Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.…

I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes, flying under the radar, or was there zero metrics being noticed for however long it took to do all that? No chart on a wall showing them the actual metrics they’re evaluating? Okay, then how about anomaly detection of any form? Taking days to perform a 90 minute task or using millions of tokens and multiple instance hand-offs on a question that other instances completed for far less time/tokens?

Consider any metric they would necessarily be tracking while it performs the evaluation. Assume they’re total assholes and only care about their marketing materials. Then narrow it down to those metrics that wouldn’t be off the scales while it discovered two zero-days in pursuit of implementing one known vulnerability. Was it the first run and they had no idea what to expect? Or did nobody notice that earlier runs completed in ### thousand tokens, but this run is off past ## million on one question, having to be compressed and handed off to new instances due to context limits, while it was in pursuit of publishing two new CVEs?

Assume they’re belligerent fools running a million benchmarks in parallel with nobody watching: there’s nothing to make them check or automatically pull the plug when it’s stuck in a loop, or did they set the threshold to 48 trillion tokens per question?

Those issues are what bugs me about it.

Post reply on HN