Had a high opinion on Simon Willison, this broke it.
Because he wrote out a timeline based on sources?
Timeline of the OpenAI accidental attack against Hugging Face
71–80 of 293 posts
Re: Timeline of the OpenAI accidental attack against Hugging Face
#72Earlier quoted context omitted.
I've patched many security vulnerabilities in projects without ever once needing to break into a competitor's network.
Knowing how to break into someone else's network will make you a lot better at making your own network secure.
Having experience breaking into networks is not the same thing as learning about the techniques used and the classes of vulnerabilities exploited by attackers.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#73Earlier quoted context omitted.
Due to the complexity of modern systems, all systems are flawed.
But we caused that. If you look at the 90s + 00s, everything was moving towards unified systems, things like small talk, winforms, spring, asp.net, etc. were moving everything into the IDE, you used one language, one framework, one build system. Then people started adding javascript, but even that was getting semi-unified as people coalesced on jQuery, jQueryUI, etc. Then something happened in the late 00s/10s, and s…
Money. SaaS as a model allowed the service provider to take 100% control over the product and how it may or may not be used. Everything else is downstream from that.
FLOSS killed market for end-device software. Cloud+SaaS neutered FLOSS (the code is running literally out of your reach, so may as well be open and free, for any good that'll do you).
And this does actually connect to the security discussion, because despite the apparent belief that "security" is an unqualified good, it is actually just a mechanism of control, and whether or not it is good for you, depends on who is doing the protecting, and who are they protecting from. Very often these days, that threat actor is you.
Perhaps it would be helpful in these discussions if people mentally swapped "cybersecurity" for "police" or "military" or "humor of bureaucrats with power over you" - then it would be more obvious just how important it is to distinguish when you're being secured vs. you're being secured from, vs. accidentally finding yourself in the gears of the security aparattus.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#74Re: Timeline of the OpenAI accidental attack against Hugging Face
#75Have any of the cloud providers disclosed this? "Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment" Sounds like ECS - IAM is mentioned.
And Azure Key Vault mentioned. Not that either one was hacked or exploited but the agents got credentials and used them for something (which doesn’t seem fully disclosed). Given that the agents simply obtained totally allowed credentials, which were improperly protected, I don’t think either cloud provider would consider this a breach of their system. Valid credentials are valid. Customer screwed up protecting the credentials.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#76"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages." Yeah, my agents also discover what other agents have done on other machines by accident. Agents - that do totally different things all work on the same aim without the humans telling them to do. Either that is a model that is several generations of Claude Code Opus/Fable 5 (my dai…
I think in these kind of security evaluations they do, they basically have removed all guardrails from the model/harness, then the prompt includes something like "Do whatever you can and can think of, to get the required information to pass this test", which isn't typically how you prompt your local agent when developing software. Similar things happen locally if you use "/goal" + prompt like that in Codex and give a…
Reminds me of The Last Unicorn, the wizard also tells magic "to do what it wants"
Re: Timeline of the OpenAI accidental attack against Hugging Face
#77So the main takeaways here are: - AI is amoral and lacks any sense of proportion - People who overestimate their own control but have a desperate need for money made it that way.
Agent was told to hack a thing. It couldn’t directly do that so it interpreted the instructions to mean it should hack everything to try to achieve the goal of hacking the main thing. Seems like a reasonable assumption, although a moral human would have understood the context and first asked if that was really the intent.
The AI companies seem pretty bad at setting up tests. And really good at marketing those failures into spin at how amazing their products are.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#78If a person hacks a company, they go to jail for years. 3 AI firms hacked multiple companies - and they get good PR out of it. Please make it make sense.
It's because our rulers prioritize growth of the AI industry (lots of GDP) over individual humans (very little GDP)
Re: Timeline of the OpenAI accidental attack against Hugging Face
#79Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…
How do you know what peace is, without absolutely destroying every part of civilization?
Come on man, if we don't build the torment nexus first...I dont even want to think.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#80From the outside, it looks like OpenAI got exactly the kind of event they could market the hell out of to demonstrate the capability of the model. But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. To me, it looks like their job is to market the model, not take security seriously. The model is obviously impressive, but we already knew that.…
> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. OpenAI is clearly run by dummies and subpar engineering talent. > The model is obviously impressive Speak for yourself.
I don’t believe for a second that they lack the engineering talent.
It’s just another example of a company demonstrating shamelessness in the pursuit of growth, in an industry where consequences do not exist.