Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

201–210 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#201

Post by JumpCrisscross »

simonw wrote: post #198
JumpCrisscross wrote: post #197

Earlier quoted context omitted.

As in was it trivially misconfigured? Would an earlier class of model have managed its way out similarly?

It wasn't that it was trivially misconfigured, it was using a piece of software (the HTTP proxy that provided access to PyPI and friends) which turned out to have a zero-day vulnerability. I don't know if earlier models would have found that vulnerability. tptacek thinks they would: https://news.ycombinator.com/item?id=49015639#49024442

Sure. I’d like independent corroboration.

It’s fair, I think, to be sceptical of OpenAI making another bout of self-serving claims. Particularly if my threshold action is changing my answer to lawmakers around whether we need reporting, licensing and potentially personal liability requirements for the engineers involved.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#202

Post by insane_dreamer »

derangedHorse wrote: post #142
cvoss wrote: post #129

The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure. Governments should immediately begin leveraging this technology on the def…

> "Use all available resources to disable the power grid of ." This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities. > Governments should immediately begin leveraging this technology on the defense side (lite…

> it would be like regulating the study of nuclear physics

not sure I agree

controlling the study of new AI model/inference algorithms would be akin to regulating the study of nuclear physics; regulating the _release_ of AI models with those capabilities would be akin to regulating uranium enrichment which allows you to put the theoretical physics to use

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#203

Post by IAmGraydon »

foobar10000 wrote: post #17
reducesuffering wrote: post #11

> It turns out relentless proactivity is the defining trait of this new generation of Mythos-class models. If you set them a goal and give them a way to get there, even inadvertently, they will figure it out. Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time... https://www.lesswrong.com/w/instrumental-convergence

Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...

Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#204

Post by lukeschlather »

dinfinity wrote: post #117
mnicky wrote: post #86

I think points that deserve more attention in the current public discourse are: - This should be a huge wakeup call for everybody. - We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. - It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox…

> The fact that it happened again seems to show their lack of ability to derive useful oversight measures. I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained . Turns out: quite good, as expected. Let me restate what I said in the other thread: Would…

My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog.

Although maybe they didn't have a watchdog agent; I definitely think using watchdogs like how Claude wraps pretty much every single tool call in Haiku to check if the command is reasonable is very necessary for unsupervised work. I suspect in the future you'll want Fable-class models wrapping every tool call, possibly with several checks "is this consistent with the goal? does it do anything unreasonably dangerous in pursuit of the goal?"

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#205

Post by ashleyn »

mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors.

Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#206

Post by allenleee »

cvoss wrote: post #129

The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure. Governments should immediately begin leveraging this technology on the def…

[flagged]

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#207

Post by simonw »

IAmGraydon wrote: post #203
foobar10000 wrote: post #17

Earlier quoted context omitted.

Or more colloquially : paperclip maximization . From OpenAI - you know, the guys who _really_ know this... Sigh... Did they finish the prompt with "And do whatever you can to get this done!" ? Cause that's the only thing that would make this even dumber...

Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?

> They created a situation that they knew would end in a cybersecurity incident.

That's a conspiracy theory.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#208

Post by simonw »

lukeschlather wrote: post #204
dinfinity wrote: post #117

Earlier quoted context omitted.

> The fact that it happened again seems to show their lack of ability to derive useful oversight measures. I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained . Turns out: quite good, as expected. Let me restate what I said in the other thread: Would…

My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog. Although maybe they didn't have a watchdog agent;…

They didn't have watchdog agents - those exist for their production models but had been deliberately removed for the purpose of this evaluation.

OpenAI wrote about how their mechanism for that in production works here: https://openai.com/index/safety-alignment-long-horizon-model...

> We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#209

Post by notaharvardmba »

ActorNightly wrote: post #184
cvoss wrote: post #129

The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure. Governments should immediately begin leveraging this technology on the def…

>The technology held by private AI companies is warfare-capable technology. This is the precisely the response OpenAI is hoping for to raise its valuation, and you fell for it. Look at it this way - whats the difference between tasking AI to break into something, versus taking a whole bunch of smart humans to do the same? The only difference is that AI is slightly easier to orchestrate. Prior to AI, there were alread…

It lowers the economic cost of a given attack, but also lowers the economic cost of protection. Not sure if it’ll be a perfect balance, but right now there’s a manufactured IMbalance due to embargos and winner picking.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#210

Post by IAmGraydon »

simonw wrote: post #207
IAmGraydon wrote: post #203

Earlier quoted context omitted.

Of course that's what they did, which is why they will never share the prompt. They created a situation that they knew would end in a cybersecurity incident. Why is the whole world acting surprised that an LLM can hack when the safety is off and it's been instructed to do so?

> They created a situation that they knew would end in a cybersecurity incident. That's a conspiracy theory.

Yes. It is. And your theory is that there was no conspiracy. What makes yours more likely than mine? You believe the people running these companies are innately good and just wouldn't do that? Are we supposed to assume that they're incapable of bad actions until proven otherwise? If so, why?
Post reply on HN