Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

31–40 of 263 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#31

Post by cogman10 »

etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

I think it's a show of these agents happily bypassing security to get stuff done.

I've actually observed similar behavior at home.

I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster.

Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi).

None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#32

Post by tosti »

skydhash wrote: post #11
frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

> where that behavior was never even intended. Strongly doubt that. Did they even share the prompt?

    C:\>CD HUGGINGF.ACE
    
    C:\HUGGINGF.ACE>DEL /F /Q *.*

Re: Timeline of the OpenAI accidental attack against Hugging Face

#33

Post by bhouston »

etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

Modern systems are complex. AI is able to thoroughly search for issues across very large surface areas. The only real way to protect will be to use AI to search for holes before other AIs find them. This type of analysis is really hard for humans to engage with successfully.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#34

Post by bwiksjdne »

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

Well to find vulnerabilities, if you can find them you can patch them. Theoretically if you find all of them you have perfectly secure software. Though it’s a double edged sword.

Goal persistence is also useful for other things like math, where it seems like there is no solution but you want the agent to keep working until it finds one.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#35

Post by KingOfCoders »

cogman10 wrote: post #31
etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

I think it's a show of these agents happily bypassing security to get stuff done. I've actually observed similar behavior at home. I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster. Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed…

" bypassing security"

If they can bypass it there is no security and the security was flawed all along.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#36

Post by mmillin »

frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

I got strong feelings of Vernor Vinge’s work here. I’m not sure how managed to come up with such a close picture to where it now seems programming and security is headed.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#37

Post by dan_q »

Meleagris wrote: post #29

From the outside, it looks like OpenAI got exactly the kind of event they could market the hell out of to demonstrate the capability of the model. But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. To me, it looks like their job is to market the model, not take security seriously. The model is obviously impressive, but we already knew that.…

> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place.

OpenAI is clearly run by dummies and subpar engineering talent.

> The model is obviously impressive

Speak for yourself.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#39

Post by [deleted] »

frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

[deleted]

Re: Timeline of the OpenAI accidental attack against Hugging Face

#40

Post by IX-103 »

skydhash wrote: post #11
frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

> where that behavior was never even intended. Strongly doubt that. Did they even share the prompt?

Did you see their presentation at Blackhat? https://youtu.be/87DyyMV0kCY?is=NnQxpOFxTX-MLu-k

They didn't share the prompt, but they did share two problematic training tasks where the AI went overboard. They also have examples from the AI's reasoning train of thought showing the AI knew it was sound something unintended.

Post reply on HN