Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

71–80 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#71

Post by Barbing »

Andes0 wrote: post #9
neitherboosh wrote: post #3

Earlier quoted context omitted.

Did you read the article you are commenting on? > There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective … To those people I say pull your heads out of the sand

unfortunately hackernews seems to have rapidly devolved, even from the point it was just a year or two ago, which wasn't a crazy bar to start. it's well on its way to just being a smaller reddit with a tighter subject range

RE: the site mention, very very last point here addresses https://news.ycombinator.com/newsguidelines.html

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#72

Post by simonw »

mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people.

I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.

I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.

I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#73

Post by milleramp »

mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

Physical guardrails do not deterministically bound cars.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#74

Post by [deleted] »

simonw wrote: post #18
protocolture wrote: post #12

>To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here! Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing…

You know this makes OpenAI look really bad , right? Hugging Face had to tell all of their users, many of them paying customers: > As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co. HF also said this, I'd be very interested to hear how that got resolved! > F…

[deleted]

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#75

Post by [deleted] »

bastardoperator wrote: post #32
Andes0 wrote: post #9

Earlier quoted context omitted.

unfortunately hackernews seems to have rapidly devolved, even from the point it was just a year or two ago, which wasn't a crazy bar to start. it's well on its way to just being a smaller reddit with a tighter subject range

I don't think it's just hacker news. I'm looking around and it seems to be widespread.

[deleted]

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#76

Post by mirashii »

simonw wrote: post #72
mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people. I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted. I expect OpenAI would agre…

> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.

I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#77

Post by simonw »

mirashii wrote: post #76
simonw wrote: post #72

Earlier quoted context omitted.

I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people. I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted. I expect OpenAI would agre…

> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project. I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be t…

> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access.

They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.

I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.

My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.

(It's also possible they were running the eval on a developer laptop somewhere!)

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#78

Post by phendrenad2 »

dinkelberg wrote: post #47
phendrenad2 wrote: post #23

Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things. The real story here is: Some people have been sounding the ala…

If a criminal can escape a prison, that's usually negligence on part of the prison staff. Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?

I can see my carefully-worded post is getting d*wnvoted, and I see from your comment why: it's being skimmed and people are assuming I'm talking about blame.

To address your point though, if every brick in the prison were made by a different person, and the prison "architects" simply glued random bricks together, I think that's closer to what we have in software right now.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#79

Post by 0xDEAFBEAD »

Eufrat wrote: post #41

Replace AI with in-development security system and does this play worse or better? An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of…

The capabilities of an OpenAI model are more general than just security. The term "AI" creates justified anxieties that this type of problem could generalize to other domains.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#80

Post by skeledrew »

gmerc wrote: post #36
simonw wrote: post #10

Important to note the actual title is "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened" - the "that happened" is important, otherwise it sounds like I think the attack was made up. Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re... > Resist th…

It can, it is both - PR spindoctoring not letting a good crisis go to waste to shape the regulatory conversation at the time the company needs it the most. Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer. Alibaba did it first ( https://georgzoeller.com/blog…

> Alibaba did it first

Hah, US frontier models 6 months behind China in cyber-security capability.

Post reply on HN