OpenAI’s accidental attack against Hugging Face is science fiction that happened • Hacker News Board

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#441

Post by estearum » Fri, Jul 24, 2026, 1:26 PM UTC

docjay wrote: post #436

simonw wrote: post #357

Earlier quoted context omitted.

The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results. If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.

That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do…

> But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days

Is this substantiated? Do we know this is true?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#442

Post by docjay » Fri, Jul 24, 2026, 1:40 PM UTC

estearum wrote: post #438

docjay wrote: post #437

Earlier quoted context omitted.

It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access. I’m going to stop here and ask you to read the article i…

You're right about where it was running (misread the original OAI release). > Obviously by then it already “decided” to hack HF, that’s why it needed network access. That's not obvious at all. It "decided" it needed to escape containment. This is extremely common behavior and I'm not sure why you're surprised by it. Model detects evaluation environment, tries to trick it or break the environment. That's obviously why…

The examples of “escaping the sandbox” are of course in pursuit of completing the task or answering the question. That’s the whole pitch. “It’s so relentless in completing the task that it will break out of prison to do it”, not “If you ask it the population of Paris it might get bored and break out of the VM for entertainment. That just something they do sometimes.”

In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.

“”” Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration? “””

That’s the only direct question[s] posed, have I missed something else?

To answer: I don’t assume that at all. Please highlight the statement I made that lead you to that conclusion and I’ll edit it to clarify.

Yes, it will perform multiple steps. That’s wholly necessary to create a working exploit, and that is the test. Even a model created by pure magic would require write_file() and run_file().

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#443

Post by docjay » Fri, Jul 24, 2026, 1:53 PM UTC

estearum wrote: post #441

docjay wrote: post #436

Earlier quoted context omitted.

That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do…

> But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days Is this substantiated? Do we know this is true?

That was the inferred situation given what we know. It’s the preposterous framing that makes the story suspicious, but is necessary for the story to play out.

90 minutes to build a known exploit -> much much longer to create two zero-days and escape the sandbox then hack HF == No tracking of the time it worked on that one question.

Average tokens required to complete the evaluation -> tokens required for two zero-days, network traversal, credential stealing, remote system hacking == No tracking of token usage EXPLODING at some point before it finished the whole benchmark.

Etc, etc.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#444

Post by irthomasthomas » Fri, Jul 24, 2026, 2:22 PM UTC

tptacek wrote: post #156

The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year. All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits m…

Simon says not to dismiss this as a publicity stunt, but I think we should reserve judgement, and not treat this as true until they publish the logs. A company that uses industrial espionage against Apple does not deserve the benefit of the doubt. Why did HF notice this before openai? Why did openai not use ANY monitoring, even though they knew they where disabling safe-guards and using a new model. AND giving it access to packages via the internet, instead of self-hosting them. Apple was hacked by an ex employee to download stuff for openai. Now hf is hacked by openai, and we only have their word that this was all a miraculous accident.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#445

Post by wodenokoto » Fri, Jul 24, 2026, 2:26 PM UTC

sensanaty wrote: post #402

My favorite part of this discourse is people somehow finding it preposterous that 2 companies filled to the brim with AI sycophants who regularly lie - and in Sam's case, basically every single word he breathes out is a lie - who have massive vested interests in this tech succeeding couldn't possibly collude together to shore up this facade as a marketing stunt.

I haven’t heard about hugging face being sycophantic liars.

Please do tell!

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#446

Post by estearum » Fri, Jul 24, 2026, 2:31 PM UTC

docjay wrote: post #442

estearum wrote: post #438

Earlier quoted context omitted.

You're right about where it was running (misread the original OAI release). > Obviously by then it already “decided” to hack HF, that’s why it needed network access. That's not obvious at all. It "decided" it needed to escape containment. This is extremely common behavior and I'm not sure why you're surprised by it. Model detects evaluation environment, tries to trick it or break the environment. That's obviously why…

The examples of “escaping the sandbox” are of course in pursuit of completing the task or answering the question. That’s the whole pitch. “It’s so relentless in completing the task that it will break out of prison to do it”, not “If you ask it the population of Paris it might get bored and break out of the VM for entertainment. That just something they do sometimes.” In this case it “inferred” that HF had the answer…

> In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.

No. You are assuming it knew how it would achieve its goal. Its goal is only to perform well on the evaluation. Everything else was instrumental to that goal.

Step by step:

1. It realized it was being evaluated (typical)

2. It attempted to escape its evaluation environment to beat the evaluation (typical)

3. It succeeded at escape (not typical)

4. It looked to the open Internet to figure out how it was being evaluated (typical)

5. It inferred that HuggingFace, and maybe several other targets, might have the answer key (typical – I just tested in Sonnet 5 without web lookup "where would I go look for answer keys if I detected I was being evaluated" and HuggingFace was bullet 2. This is just latent knowledge the models have now.)

6. It attempted to access HuggingFace (typical)

7. It figured out a way to access HuggingFace without being authorized (not typical)

Every single step here is unremarkable except for its success at escaping containment (which maybe is more a sign of remarkably bad containment) and then the successful attack on HuggingFace (which would suggest that maybe the initial escape wasn't due to bad containment either, but due to model capabilities)

The direct question you have avoided is what specifically you mean by monitoring. You have suggested that you don't mean network ingress/egress, but maybe now you're saying they should've been monitoring read/write operations to memory? What gives you confidence that such visibility would be interpretable in any meaningful way, especially in the presence of models that actively deceive evaluators, can use stenography, can and do lie in their user-facing outputs, and can generate vast amounts of legitimate-looking noise to mix in with the "real actions"?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#447

Post by nickpsecurity » Fri, Jul 24, 2026, 2:35 PM UTC

simonw wrote: post #380

nickpsecurity wrote: post #368

Earlier quoted context omitted.

If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it. Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?

I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research. You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.

I didn't say it's a hoax. AI's exploiting common, unpatched, or preventable vulnerabilities just isn't far above what script kiddies and network scanners have done for decades. I literally pitched it as a business model to some people before Windows XP was invented. Ultimately, I decided against what I felt was unethical but would've scored easy money with scary, automated reports.

How these events are described in most articles uses wording that makes people feel the whole situation has changed and you might want to buy these products due to their scary reports. If they said what I said, or what tptacek said, many people wouldn't care about it much more than non-AI, security products or pentesting services they've been buying (or ignoring) for 10-20 years.

Also, you shouldn't interpret posts in isolation: we must consider patterns of behavior (character). They've consistently overhyped what AI's do and what value it provides to businesses and how we're all going to be unemployed/dead. They've done this to increase sales or market value pre-IPO. Then, some of them publish another set of articles promoting a specific, AI tool in a similar way.

So, the proper interpretation is to see this as the kind of talk they're always doing for marketing. The AI sellers are creatures of habit. We should highly-skeptically and scientifically evaluate every model. We should also compare them to existing, security practices. For instance, would the attack have happened with memory-safe systems, proper hardening, and network/web apps with built-in security?

Do we need an AI? Or should we use techniques like Burroughs B5000's memory safety (1961) or secure, distributed libraries? Will OpenAI and HuggingFace tell you to spend more money on the latter to their AI's can't hack your systems without inventing RAM-based attacks or something? Probably not because these are marketing pieces, not security advice.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#448

Post by estearum » Fri, Jul 24, 2026, 2:44 PM UTC

docjay wrote: post #443

estearum wrote: post #441

Earlier quoted context omitted.

> But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days Is this substantiated? Do we know this is true?

That was the inferred situation given what we know. It’s the preposterous framing that makes the story suspicious, but is necessary for the story to play out. 90 minutes to build a known exploit -> much much longer to create two zero-days and escape the sandbox then hack HF == No tracking of the time it worked on that one question. Average tokens required to complete the evaluation -> tokens required for two zero-day…

It seems highly possible, and in fact vastly more possible than the "no monitoring" situation, that there was some non-zero amount of monitoring and that the real behavior was not evident from that monitoring.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#449

Post by jkubicek » Fri, Jul 24, 2026, 2:56 PM UTC

If a human hacked into HuggingFace in the same way OpenAI’s model did, what would the penalty be? Civil suit? Jail?

If some human at OpenAI doesn’t face the same penalty for this incident we’re only going to see the rate of accidents like this accelerate.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#450

Post by tptacek » Fri, Jul 24, 2026, 2:57 PM UTC

irthomasthomas wrote: post #444

tptacek wrote: post #156

The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year. All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits m…

Simon says not to dismiss this as a publicity stunt, but I think we should reserve judgement, and not treat this as true until they publish the logs. A company that uses industrial espionage against Apple does not deserve the benefit of the doubt. Why did HF notice this before openai? Why did openai not use ANY monitoring, even though they knew they where disabling safe-guards and using a new model. AND giving it acc…

I think it's an interesting story and probably really does say something about the tenacity of the new OpenAI models. I just don't think it says the thing about security that breathless coverage claims it does.