Wintermute is out there…
Timeline of the OpenAI accidental attack against Hugging Face
221–230 of 440 posts
Re: Timeline of the OpenAI accidental attack against Hugging Face
#222Earlier quoted context omitted.
The way I read it was different things happening over several runs, such as the agents comparing notes so to speak, using artifactory
I can’t get over how the process is exactly what a hacker hive does. Communicate leaving notes in some random file.
I can’t get over that no one noticed any of this going on at OpenAI.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#223All of the latest developments surrounding these attacks are actually a really bad sign for these labs. It seems that raw intelligence of frontier models has largely plateaued (despite what is basically an order of magnitude increase in parameter size) so to make any significant improvements and to justify massive capex spend they have resorted to reinforcement training models to never give up and brute force the sea…
> I wonder if we can still trust these labs to develop something that benefits humanity
Surely soon they'll comprise only people who are blind to the inevitable danger and people who don't care about it. Because who else would feel at all comfortable doing the job?
Re: Timeline of the OpenAI accidental attack against Hugging Face
#224Why are the agents trying so hard to communicate with each other, leaving messages and so on?
We are assigning semantics to systems that deal only in syntactics. The entire problem with the current "AI" hype is squarely based on how we interpret output from systems based on statistical modelling of natural language. That software is built on top of human language and these systems can be used for uncanny automation is a huge societal problem at the moment because we are all assigning meaning to patterns that…
I enjoyed this rant.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#225Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…
It's a war.
And because of that we are a few steps away from WarGames [1]
Re: Timeline of the OpenAI accidental attack against Hugging Face
#226If a person hacks a company, they go to jail for years. 3 AI firms hacked multiple companies - and they get good PR out of it. Please make it make sense.
It's because our rulers prioritize growth of the AI industry (lots of GDP) over individual humans (very little GDP)
Anyway Google Peter Thiel Dialog
Re: Timeline of the OpenAI accidental attack against Hugging Face
#227Earlier quoted context omitted.
How can it escape an "install package " proxy? reducePrivs() serve get(package) { secPackage = secure(package) getBinaryFromArtifactory(secPackage) } I would think the code is very small and easier to verify, it doesn't especially have the ability to write files and act as a message board as Artifactory did. And even if the agent tries to hack that, the attack surface is 1000x smaller and the possibility also much sm…
I mean, it's just the same problem. The machine still has Internet access. It doesn't need to. The entire package manager repository could just be in an offline cache. They don't need Internet to give their agents access to tons of software.
"They don't need Internet to give their agents access to tons of software."
I think that was the requirement, but yes, the cache could have been offline.
Still then they could have hacked it to create the message boards - but not use it to access the internet.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#228I wish we could stop sensationalizing this about the AI and really just understand the incompetence of the labs disabling an internet connection in a sandbox.
As written it sounds like you're saying that it was incompetent of the labs to disable the sandbox internet access? They tried to disable open internet access but the models zero-day'd their Artifactory package registry and got internet access anyway. No sensation... that's just what happened.
If you really wanted to sandbox a machine you’d offline cache the packages and not give it any physical route to the internet, not via a jump box, not via a proxy, nothing.
This was poorly executed.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#229Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
I think it's a show of these agents happily bypassing security to get stuff done. I've actually observed similar behavior at home. I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster. Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed…
It’s unpredictable when it decides to bypass though.
Security by obscurity is pretty useless against people and ai that are smarter than us.
Re: Timeline of the OpenAI accidental attack against Hugging Face
#230Earlier quoted context omitted.
We are assigning semantics to systems that deal only in syntactics. The entire problem with the current "AI" hype is squarely based on how we interpret output from systems based on statistical modelling of natural language. That software is built on top of human language and these systems can be used for uncanny automation is a huge societal problem at the moment because we are all assigning meaning to patterns that…
I enjoyed this rant.
[dead]