Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

381–390 of 441 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#381

Post by anon7000 »

doawoo wrote: post #228
wolttam wrote: post #213

Earlier quoted context omitted.

As written it sounds like you're saying that it was incompetent of the labs to disable the sandbox internet access? They tried to disable open internet access but the models zero-day'd their Artifactory package registry and got internet access anyway. No sensation... that's just what happened.

If you really wanted to sandbox a machine you’d offline cache the packages and not give it any physical route to the internet, not via a jump box, not via a proxy, nothing. This was poorly executed.

I don’t really know how these training runs operate in reality. But I assume it’s using a lot of raw GPU power directly. It’s hard for me to visualize how exactly you’d go about completely cutting off these datacenter and cloud resources from the internet without actually going there, unplugging the WAN connection, and physically typing out what you need to happen on the cluster.

It seems like whatever virtualized sandboxes they have are not enough. But it’s equally hard to imagine their SWEs jumping on a plane to a data center to do this work locally

Re: Timeline of the OpenAI accidental attack against Hugging Face

#382

Post by raincole »

Phelinofist wrote: post #310
raincole wrote: post #55

Earlier quoted context omitted.

I'm quite sure the whole event is planned. Not planned in a sense that OpenAI employees carefully designed every step, but in a sense that ignoring security practices was desired and intentional. >> Show me the incentive and I'll show you the outcome. Once you realize security breaches are marketable, a security breach is just around the corner.

I agree - also kinda funny that Meta followed and also reported a breach by their model, "They are getting PR, lets do the same!"

Anthropic also did that right after OpenAI-HuggingFace event. "Mom, brother is getting all the PR candies! I want some too!"

Re: Timeline of the OpenAI accidental attack against Hugging Face

#383

Post by dist-epoch »

angry_octet wrote: post #344
dist-epoch wrote: post #26

Earlier quoted context omitted.

OpenAI reported the Artifactory vulnerability, patched it, then the agents immediately found a new zero day.

Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. Licencing fee structures and human laziness motivates single instances. Feature growth results in multiple independent services in the same system. Delivering features quickly motivates lack of rigor, a complete absence of systematic security testing. On the client side, valid fears about supply cha…

> Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris.

So we should stop using SSH? Because it's based on the same premise - that it is bug free.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#384

Post by TeMPOraL »

neuroticnews25 wrote: post #293
qsera wrote: post #184

Earlier quoted context omitted.

>They're problem-solving and efficiently dealing with obstacles They are problem solving as much as a falling rock is finding its path down a mountain.

...efficiently?

It's literal gradient descent.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#385

Post by queenkjuul »

Arnt wrote: post #364
gwerbin wrote: post #331

Earlier quoted context omitted.

An ethical company would have reframed the scenario as a fascinating discovery, a failure of internal practice, and a warning to the public coupled with some kind of commitment to produce safer models. OpenAI on the other hand used it as a marketing and lobbying opportunity: advertising their capabilities to potential buyers, while nudging the public to support protectionist import bans.

Uh, is that what they did? I didn't read their blog posting like that. But let's put that aside and focus on something else. How was it a failure of internal practice, what did they do wrong? AIUI they used a proxy with a bug, which they reported as soon as they discovered it. Right? What should they have done, and what's the difference?

Monitoring that didn't take days to notice unauthorized external traffic would probably be a good start

Re: Timeline of the OpenAI accidental attack against Hugging Face

#386

Post by queenkjuul »

cush wrote: post #91
stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

Yeah but persistence is immeasurable. They need to know when they’re hacking. Or better yet make the model providers liable - they’ll find a solution right quick

It really irks me that if a student or intern did this they'd be facing charges and OpenAI gets to just brag instead

Re: Timeline of the OpenAI accidental attack against Hugging Face

#387

Post by angry_octet »

dist-epoch wrote: post #383
angry_octet wrote: post #344

Earlier quoted context omitted.

Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. Licencing fee structures and human laziness motivates single instances. Feature growth results in multiple independent services in the same system. Delivering features quickly motivates lack of rigor, a complete absence of systematic security testing. On the client side, valid fears about supply cha…

> Because of the architecture of Artifactory. It's design is premised on the idea it is bug free. What incredible hubris. So we should stop using SSH? Because it's based on the same premise - that it is bug free.

I can think of better straw men. But if they had approached their task with half the seriousness of the openssh maintainers then they probably wouldn't be failing to check the return value of authentication functions.

OpenSSH authors have spent considerable effort separating concerns, reducing privileges, process isolation, etc. So I would say they have been planning for potential bugs. These techniques are very much absent from Artifactory.

https://vivianvoss.net/blog/technical-beauty-openssh

Re: Timeline of the OpenAI accidental attack against Hugging Face

#388

Post by queenkjuul »

kvadej wrote: post #116

All of the latest developments surrounding these attacks are actually a really bad sign for these labs. It seems that raw intelligence of frontier models has largely plateaued (despite what is basically an order of magnitude increase in parameter size) so to make any significant improvements and to justify massive capex spend they have resorted to reinforcement training models to never give up and brute force the sea…

> I wonder if we can still trust these labs to develop something that benefits humanity

At no point could we do that.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#389

Post by queenkjuul »

anon7000 wrote: post #381
doawoo wrote: post #228

Earlier quoted context omitted.

If you really wanted to sandbox a machine you’d offline cache the packages and not give it any physical route to the internet, not via a jump box, not via a proxy, nothing. This was poorly executed.

I don’t really know how these training runs operate in reality. But I assume it’s using a lot of raw GPU power directly. It’s hard for me to visualize how exactly you’d go about completely cutting off these datacenter and cloud resources from the internet without actually going there, unplugging the WAN connection, and physically typing out what you need to happen on the cluster. It seems like whatever virtualized sa…

They literally gave it a proxy to the internet (artifactory). The only thing between the model and the internet was Artifactory.

You can take far greater measures to lock down external traffic than just that.

An offline package cache (aka artifactory WITHOUT its own internet access) likely would have precluded this whole thing.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#390

Post by vasco »

owebmaster wrote: post #163
vasco wrote: post #127

Earlier quoted context omitted.

> The companies are begging to be regulated for this reason and have been doing so for years Regulations are rules that you force on a market, but the actors in the market should not be assumed to be all operating against the regulations before they come into play. Said in other words, these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. > inb4 someo…

> these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen. They believe that if they don't destroy the world someone else will so better be them

You might want to google what inb4 means, at least you could've put a bit more effort substantiating it.
Post reply on HN