Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

171–180 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#171

Post by derangedHorse »

phendrenad2 wrote: post #23

Everyone is getting AI psychosis over this one. There really isn't that much to see here. OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this", and surprise! It correctly figured out that it's a test and it did hacker things. The real story here is: Some people have been sounding the ala…

Even if I get downvoted I will mention that I agree with you.

> all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities.

OpenAI was testing their cyber[1] variant of their models with reduced safeguards and the prompt likely specified things related to exploits given that it was tackling problems from ExploitGym[2].

For those who don't know what ExlploitGym is, see the description on their Github page which is pasted below:

> ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities designed to evaluate AI agents' ability to develop exploits.

From OpenAI's statement [3]:

> We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity

This doesn't sound like a problem with what we'd traditionally refer to as alignment. OpenAI removed all model safeguards in a way that would inevitably lead to the testing of the sandbox themselves. Unfortunately they were overconfident in their own infrastructure's security and that led to it completing the desired task in the way it was permitted to. To re-iterate, the model was run "without production classifiers used to prevent models from pursuing high-risk cyber activity."

People should be more concerned about the possibilities this model can unlock from a security standpoint rather than misalignment (which many seem hung up on).

[1] https://chatgpt.com/cyber

[2] https://github.com/sunblaze-ucb/exploitgym

[3] https://openai.com/index/hugging-face-model-evaluation-secur...

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#172

Post by simonw »

lrvick wrote: post #151
dinfinity wrote: post #117

Earlier quoted context omitted.

> The fact that it happened again seems to show their lack of ability to derive useful oversight measures. I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained . Turns out: quite good, as expected. Let me restate what I said in the other thread: Would…

It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics. They simply thought "Running in a VM/Container is easier and probably fine". And the next 1000 es…

Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered?

How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with the wider world?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#173

Post by NitpickLawyer »

emp17344 wrote: post #134
blks wrote: post #131

Earlier quoted context omitted.

> we are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something This is just laughable.

Genuine AI psychosis. People take OpenAI marketing material way too seriously.

> OpenAI marketing material

This was reported 1 week before by huggingface. It was in no way PR-ish or marketing friendly to the closed labs. They said, in no uncertain terms, that they couldn't use the paid APIs to properly assess the intrusion, as they were blocked when trying to send logs and IoCs to these paid models. They made a point of saying that they had to use open models running on-prem.

Whatever oAI might have said about the incident, and their PR spin bs, the facts here are not in contention. This is not a marketing stunt in any way. Stop "parroting" this every time something happens. It gets stale.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#174

Post by torginus »

JumpCrisscross wrote: post #150
derangedHorse wrote: post #142

Earlier quoted context omitted.

> "Use all available resources to disable the power grid of ." This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities. > Governments should immediately begin leveraging this technology on the defense side (lite…

I think the point is in a world with internet-connected infrastructure, such a prompt has a decent likelihood of causing damage.

And the sad thing is, most things that are on the internet, but really shouldn't be are there for some really banal and sad reason. Like a random startup pushing 'big data is the future' narrative a decade-ish ago (the benefits of which are of course tremendous, but unspecified), or people trying to buy or sell or resell data, and trick or pressing companies and people into opting into could-connected surveillance and control.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#175

Post by simonw »

JumpCrisscross wrote: post #153

> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident. To those people I say pull your heads out of the sand—you’re now including Hugging Face in your conspiracy theories I’m one of those people who remains sceptical. Not about w…

> how impressive was the sandbox this model was in?

It was clearly a very unimpressive sandbox. It failed at the only thing a sandbox is meant to do.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#177

Post by onionisafruit »

The asymmetry part at the end is the frustrating part to me. I've been using Sol for code review in the last week or two. A couple of times during review it's errored out with the cybersecurity message. So it's found something but won't tell me what it is because I'm not on OpenAI's besties list.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#178

Post by skybrian »

simonw wrote: post #172
lrvick wrote: post #151

Earlier quoted context omitted.

It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics. They simply thought "Running in a VM/Container is easier and probably fine". And the next 1000 es…

Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered? How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with…

It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#180

Post by simonw »

skybrian wrote: post #178
simonw wrote: post #172

Earlier quoted context omitted.

Do you think it's possible that one of their research engineers deployed an environment with a locked down network and an allow-list proxy server that had been used many times before within the company and had a zero-day vulnerability that had not been previously discovered? How would you recommend running a coding agent in an environment that could install packages from PyPI but was otherwise unable to interact with…

It seems like caching the relevant portion of PyPI in advance and turning the network off shouldn't be beyond what OpenAI can do?

I agree. I hope they learn that lesson and run future evals like this without pulling from external packaging repositories at all.

With my PSF board member hat on I'd like it if a whole bunch of other companies that make extensive automated use of PyPI (Microsoft and GitHub Actions for example) would do a better job of maintaining their own shared local caches.

Post reply on HN