Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

191–200 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#191

Post by tptacek »

efromvt wrote: post #186
tptacek wrote: post #156

The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year. All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits m…

Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.

I think the mental model people have about this is that pre-AI there were humans picking individual targets and post-AI the computer itself randomly picks targets. But you get the same unexpected collateral damage outcome when a human misconfigures a decent pentest tool.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#192

Post by verdverm »

simonw wrote: post #164
danjc wrote: post #157

The part I don't understand is how a model can break out of a sandbox. It should be fairly easy to make a sandbox that sandboxes.

The sandbox still allowed the agents to install additional dependencies (from PyPI etc) that they needed. It did this by locking down all network access with the exception of an HTTP proxy that only allowed read access to PyPI and a few other places. This is a very common pattern. I wrote about how OpenAI were doing this for their production ChatGPT container environment (using Artifactory) back in January: https://s…

> This is a very common pattern.

Sounds like a bad habit for security testing Ai. It's not that hard to build an internal mirror and proxy that, keeping the real internet physically separated if needed, and truly locked down if concerns aren't as great.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#193

Post by simonw »

torginus wrote: post #179

> A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. Did huggingface get pickled?

I like how this article put it: https://martinalderson.com/posts/huggingface-openai-exploit/

> A final point on this - Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.

But yeah, from the way HF described it a Python pickle hole looks possible. Their datasets library uses the pandas.read_pickle() method here: https://github.com/huggingface/datasets/blob/d21c5816d5d1961...

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#194

Post by rurban »

bethekidyouwant wrote: post #133
rurban wrote: post #107

Earlier quoted context omitted.

Nonsense. Huggingface reported it to police!

What does calling the police prove? (nothing i hope)

It proves that it was certainly not a sama PR stunt, as many cynics are alleging.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#195

Post by mnicky »

rybosome wrote: post #160
mnicky wrote: post #86

I think points that deserve more attention in the current public discourse are: - This should be a huge wakeup call for everybody. - We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. - It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox…

Very much agreed on the significance. The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur. My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs. All of the systems which w…

The air gap would probably help and after this incident I hope labs will think about using such a measure when appropriate.

On the other hand I think that proper solution for these kinds of problems is not at a sandbox level, but at a model alignment level.

Also it shows that maybe the most serious risk comes not from releasing models publicly but from internal, pre-release period where you sometimes need/want to lift some guardrails a bit etc.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#196

Post by verdverm »

gilbetron wrote: post #123

This makes me think: what are the odds weights from Frontier Models have already been stolen? Something like Mythos without "guardrails" seems like a hell of a glittering gem for many nefarious actors.

With Kimi 3 weights coming, and China committing to the open weight ethos, everyone will be able to download frontier models. Running them is another problem.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#197

Post by JumpCrisscross »

simonw wrote: post #175
JumpCrisscross wrote: post #153

> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident. To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories I’m one of those people who remains sceptical. Not about w…

> how impressive was the sandbox this model was in? It was clearly a very unimpressive sandbox. It failed at the only thing a sandbox is meant to do.

As in was it trivially misconfigured? Would an earlier class of model have managed its way out similarly?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#198

Post by simonw »

JumpCrisscross wrote: post #197
simonw wrote: post #175

Earlier quoted context omitted.

> how impressive was the sandbox this model was in? It was clearly a very unimpressive sandbox. It failed at the only thing a sandbox is meant to do.

As in was it trivially misconfigured? Would an earlier class of model have managed its way out similarly?

It wasn't that it was trivially misconfigured, it was using a piece of software (the HTTP proxy that provided access to PyPI and friends) which turned out to have a zero-day vulnerability.

I don't know if earlier models would have found that vulnerability. tptacek thinks they would: https://news.ycombinator.com/item?id=49015639#49024442

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#199

Post by lrvick »

simonw wrote: post #172
lrvick wrote: post #151

Earlier quoted context omitted.

It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics. They simply thought "Running in a VM/Container is easier and probably fine". And the next 1000 es…

Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered? How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with…

Models started proving readily capable of autonomously finding 0days over a year ago. At that point (ideally long before) the responsible sandbox for research labs with new models highly capable of this sort of thing should be literally airgapped. Disconnect the rack physically from any access to the internet, and give it an offline mirror of all of PyPi, NPM, Wikipedia, whatever as needed. They can afford the disk space. They already mirror it all anyway as training data. That is the bare minimum. Personally I would rule out side channel attacks with a $40k Tempest spec rack for good measure. Can interact with it via pubsub over a fiber optic serial terminal.

Such things are pretty normal in biotech, fintech, and defense who must have 0days in their threat models. AI labs absolutely need to start being forced to operate this way by endless negligence lawsuits. Move fast and break things culture is not going to cut it anymore.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#200

Post by verdverm »

joshstrange wrote: post #188
verdverm wrote: post #185

Earlier quoted context omitted.

Why bother when more companies than not make their weights available on Huggingface?

I was thinking specifically about the frontier lab models (OpenAI/Anthropic/Google)

We'll have ~Fable level weights in the coming days (K3). Look out to the end of the year and there will be multiple options. There are seemingly more frontier labs than the three US ones.
Post reply on HN