Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

51–60 of 293 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#51

Post by TeMPOraL »

cogman10 wrote: post #31
etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

I think it's a show of these agents happily bypassing security to get stuff done. I've actually observed similar behavior at home. I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster. Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed…

I don't now, I emphasize with the agent here. The experience of modern computing is largely that of a computer standing between you and your goal and being obnoxious. This holds true for both normies in their daily consumption, and software people deep at work. An agent that has no skill or no willingness to bludgeon through "the computer says no" is not very useful.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#52

Post by KingOfCoders »

"The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face."

Why, what was the prompt?

I told Claude today to wire plugins on Linux into a sound pipeline to remove noise. Did some astonishing things, played sound through the pipeline, measured it etc. I told it to optimize my sound for TF2 and it played the spy_decloak samples, measured them and made them easier to hear, astonishing too.

But it did not go to hack Amazon because it could.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#54

Post by KingOfCoders »

detourdog wrote: post #48
KingOfCoders wrote: post #43

"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages." Yeah, my agents also discover what other agents have done on other machines by accident. Agents - that do totally different things all work on the same aim without the humans telling them to do. Either that is a model that is several generations of Claude Code Opus/Fable 5 (my dai…

The agents sound like old school hackers that would just explore what access they could gain. Creating a file for other hackers and themselves. The fact that there were 3 events for 3 major players does make it seem co-ordinated.

My read is: One did it as a PR stunt, the others saw that every media reported on this and did the same.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#55

Post by raincole »

Meleagris wrote: post #29

From the outside, it looks like OpenAI got exactly the kind of event they could market the hell out of to demonstrate the capability of the model. But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. To me, it looks like their job is to market the model, not take security seriously. The model is obviously impressive, but we already knew that.…

I'm quite sure the whole event is planned. Not planned in a sense that OpenAI employees carefully designed every step, but in a sense that ignoring security practices was desired and intentional.

>> Show me the incentive and I'll show you the outcome.

Once you realize security breaches are marketable, a security breach is just around the corner.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#56

Post by alansaber »

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

They'll set up guardrails but I believe the point is better code uae / better long running tasks > inevitable that cyberattacks will be easier

Re: Timeline of the OpenAI accidental attack against Hugging Face

#57

Post by alansaber »

frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

Given the amount of raw compute going into models it would be more surprising if we couldn't get events like this

Re: Timeline of the OpenAI accidental attack against Hugging Face

#58

Post by detourdog »

KingOfCoders wrote: post #54
detourdog wrote: post #48

Earlier quoted context omitted.

The agents sound like old school hackers that would just explore what access they could gain. Creating a file for other hackers and themselves. The fact that there were 3 events for 3 major players does make it seem co-ordinated.

My read is: One did it as a PR stunt, the others saw that every media reported on this and did the same.

or they were scared and figured this was the right time to reveal.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#59

Post by alansaber »

dan_q wrote: post #19
frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

> This feels straight out of sci-fi. Most AI marketing is straight up science fiction.

Fake it til you make it

Re: Timeline of the OpenAI accidental attack against Hugging Face

#60

Post by namdnay »

mmillin wrote: post #36
frays wrote: post #7

This feels straight out of sci-fi. We're talking about AI agent swarms emergently coordinating over the span of weeks and pulling off sophisticated strategies under adversity in an environment where that behavior was never even intended. Anyone brushing this off as just a "bad prompt" is completely missing the scale of what actually happened.

I got strong feelings of Vernor Vinge’s work here. I’m not sure how managed to come up with such a close picture to where it now seems programming and security is headed.

I reread a deepness recently, and it’s funny how the “focused” (and more importantly, how they are used) mirror LLMs
Post reply on HN