Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

51–60 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#51

Post by gr_norm »

simonw wrote: post #18
protocolture wrote: post #12

>To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here! Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing…

You know this makes OpenAI look really bad , right? Hugging Face had to tell all of their users, many of them paying customers: > As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co. HF also said this, I'd be very interested to hear how that got resolved! > F…

I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade.

It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.

For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#52

Post by simonw »

gmerc wrote: post #28
simonw wrote: post #22

Earlier quoted context omitted.

It's the first report I've seen of a model both escaping a sandbox and then actively exploiting another company, when neither of those actions was intended.

It sure isn’t. https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to... There’s also daily reports from people that have these models escape docker, which happens regular enough that it would be considered negligence to use docker as sandbox.

That Alibaba one was a sandbox escape (I agree those are common) but what's new with the OpenAI story is an attack against another company.

This wasn't a small attack either, Hugging Face published a security advisory for their users while they were still figuring out what happened.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#53

Post by rubyfan »

I can’t help but feel all warm and fuzzy with my head in the sand and getting a shout out in TFA for calling it marketing.

We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.

The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#54

Post by windexh8er »

simonw wrote: post #35
windexh8er wrote: post #31

Earlier quoted context omitted.

So this is either shitty OpSec or this is yet more marketing spin to ramp back FUD to 11 again. If it's the latter I'm imagining Dario told Sam that it's their turn this time. Aligns with the premise that this is straight out of science fiction.

It's bad OpSec by the research team. Their sandbox was not bulletproof and their monitoring was insufficient. It looks to me like their production models have a lot more monitoring than their research clusters.

I also love how clear a picture your piece paints that these highly capable models are as useful as a rock when it comes to a defender role. The line is too fine, even for Mythos. Irony.

But to have an open weights Chinese model come to the rescue for HF is the cherry on top! If there wasn't a very pointed example of why gating models was a very bad thing previously, well - here we are.

Also, this sounds interesting but there are only a few that can pull this type of heist off currently. And those are the people who are gating the models / have access to large AI DCs. Because, I can only assume this test burned tokens easily within the 7 figure and possibly even 8 figure levels (subsidized market rate costs). This won't / can't happen outside of frontier labs or nation states currently. Yet we should all be worried about Mallory equipped with her OpenRouter account.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#55

Post by simonw »

gr_norm wrote: post #51
simonw wrote: post #18

Earlier quoted context omitted.

You know this makes OpenAI look really bad , right? Hugging Face had to tell all of their users, many of them paying customers: > As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co. HF also said this, I'd be very interested to hear how that got resolved! > F…

I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade…

Here's the language that makes me think they are taking this seriously:

> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.

That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.

I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#56

Post by dang »

QGQBGdeZREunxLe wrote: post #38
Bawoosette wrote: post #5

The title (currently "OpenAI's accidental cyberattack against Hugging Face is science fiction") suggests some information had been hidden that makes the incident less significant than claimed. The article argues the opposite, and the last two words of the full title are "that happened."

Looks like it's been edited now and makes more sense "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"

I've edited it now but had to de-cyber cyberattack because the limit is 80 chars.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#57

Post by bubble_niter »

dang wrote: post #56
QGQBGdeZREunxLe wrote: post #38

Earlier quoted context omitted.

Looks like it's been edited now and makes more sense "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"

I've edited it now but had to de-cyber cyberattack because the limit is 80 chars.

Thanks to dang for restoring the few missing words that separate a factual title from a promotional ambush

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#58

Post by gnabgib »

dang wrote: post #56
QGQBGdeZREunxLe wrote: post #38

Earlier quoted context omitted.

Looks like it's been edited now and makes more sense "OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"

I've edited it now but had to de-cyber cyberattack because the limit is 80 chars.

This specific promo machine does not need your assistance :/

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#59

Post by windexh8er »

simonw wrote: post #55
gr_norm wrote: post #51

Earlier quoted context omitted.

I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade…

Here's the language that makes me think they are taking this seriously: > We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete. That's no…

> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.

With who? Who are these "defenders"? None of the US labs have done much for the greater good as of... Ever. Of course a frontier provider can leverage their own resources at scale and pull something like this off. If anything this should showcase how dangerous OpenAI and Anthropic are in their current states and maybe the powers shouldn't be concentrated as they continue to move.

I will bet that the RCA debriefed by OAI is going to be a lot of lipstick and very little meat.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#60

Post by charcircuit »

simonw wrote: post #22
charcircuit wrote: post #21

This isn't the first time a model has escaped a sandbox. And models trying to find alternate routes to do something when one route is blocked is nothing new.

It's the first report I've seen of a model both escaping a sandbox and then actively exploiting another company, when neither of those actions was intended.

How many other people would publicize that they hacked into another company assuming they even noticed?

>when neither of those actions was intended.

It was a single goal that it didn't give up easily on.

Post reply on HN