Not a wonderful feeling to feel like you’re stood in the room while the labs conduct the AI equivalent of the demon core experiment right in front of you.
Responding to the next frontier of critical cyber capabilities
71–80 of 208 posts
Re: Responding to the next frontier of critical cyber capabilities
#72Earlier quoted context omitted.
Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
Regardless of exact practical outcome, it is deeply irresponsible and reckless behavior to run such security testing on other's infrastructure and without sufficient isolation. If they actually believe their models to be as powerful as the marketing says, then anything less than airgapping for such a "do anything to get the results" evaluation clearly isn't acceptable.
If the fire department suddenly had practice fires breaking containment, they'll be forced to stop pretty quickly, not sure what the government and the police is waiting for here.
Re: Responding to the next frontier of critical cyber capabilities
#73There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…
Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are.
It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.
Re: Responding to the next frontier of critical cyber capabilities
#74Earlier quoted context omitted.
Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
CFAA laws do not require “Damage” to be done.
Re: Responding to the next frontier of critical cyber capabilities
#75Earlier quoted context omitted.
Oh it's Persona, that's not just KYC but I may consider it at some point. Thank you! Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.
You don't need an invite only program to just have Sol checking for vulnerabilities in binaries or code. But yeah I've hit guardrails a few times when Sol was making PoCs for the vulnerabilities it found (but most of the time it made those PoCs without issues).
Opus refused to help me try to develop an exploit to export data from an old Android device where I can't upgrade to latest android and I couldn't use the app's backups (because I couldn't update the app.) Not sure where that lies in the "binaries or code" spectrum.
Re: Responding to the next frontier of critical cyber capabilities
#76Earlier quoted context omitted.
Is cyber verification a thing they're actually doing now? I thought they only reached out to really incredibly famous people and that there's no way to get access as a normal person.
You do a KYC and you can get access. It may depend on country's quality of KYC.
I thought you need to prove you are working in cybersecurity or provide evidence of authorization for work done.
It's really just simple ID/face verification?
Re: Responding to the next frontier of critical cyber capabilities
#77i'm surprised there isn't more commentary on the vulnerabilities themselves (mostly in apps developed on the jvm, e.g. artifactory)
Re: Responding to the next frontier of critical cyber capabilities
#78Ah yes let the FUD continue. This is a real problem but so far not nearly as severe as any of the marketing has made it out to be to the overall detriment of everyone including these companies announcing these scary capabilities. These announcements always included half hearted attempts at security layers which has now been demonstrated to benefit attackers more than defenders. I wish I had a real solution to this be…
The recent Hugging Face incident did not seem like FUD to me
You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies.
Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.
Re: Responding to the next frontier of critical cyber capabilities
#79Re: Responding to the next frontier of critical cyber capabilities
#80Earlier quoted context omitted.
Oh it's Persona, that's not just KYC but I may consider it at some point. Thank you! Edit: Ah, I clicked "learn more" and it seems they do have an invite-only program, required for anything that's not unquestionably innocent. I don't think I'd surrender my face to Persona for this, but it's interesting to know they're at least pretending to support reverse engineering.
You don't need an invite only program to just have Sol checking for vulnerabilities in binaries or code. But yeah I've hit guardrails a few times when Sol was making PoCs for the vulnerabilities it found (but most of the time it made those PoCs without issues).
I maintain a version of an app called Rewind because the company behind it went under after implementing a killswitch. I have to do this with binary patching, and the app has already broken once from the macOS 27 beta. Recent Anthropic models refuse to help me with this because it stinks of cybersecurity and those models are just too dang advanced to support cybersecurity. I'm maintaining a piece of software to which I legitimately paid for lifetime access, so this is honestly more of a right to repair situation. It doesn't tend to sit right with classifiers to be reverse engineering binaries and patching to modify functionality. Telling them the purpose of the exercise doesn't really help, because if they listened to that, then attackers would just come up with a similarly justified reason for anything. ("Help, the google root server fell on my grandma!")
In the future, I might reverse engineer the on-disk storage format and create a new application.