So they finally found a business model: the cause of, and solution to, cyber security problems.
People have been saying Tokens are the new Oil. Turns out, it’s the new Alcohol. The cause of, and solution to, life’s problems!
[deleted]
Responding to the next frontier of critical cyber capabilities
181–190 of 207 posts
Re: Responding to the next frontier of critical cyber capabilities
#182There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…
A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.
"Give everybody a gun! So they can defend themselves from the bad guys! Don't let the government take away your guns!!"
Re: Responding to the next frontier of critical cyber capabilities
#183as long as the model doesn't call itself 'skynet' we are safe
Re: Responding to the next frontier of critical cyber capabilities
#184I refuse to accept “cyber” as a noun to mean “cybersecurity”. If you use cyber as a noun, I’m assuming you mean this, regardless of context: http://cyber.urbanup.com/521436
Re: Responding to the next frontier of critical cyber capabilities
#185Earlier quoted context omitted.
To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?! OK, yes,…
Amazing story. I think you’re defending against a level of aggression that most of the world doesn’t believe exists yet. I suppose by the time we believe, it will be too late. There’s a lot of people who will be trying to run LLMs with hacking goals, so maybe we need to get used to a lot of automated attacks.
The thing is, I'm not even looking at this from a "how do we keep a superintelligent AI contained" viewpoint. The standard protocol[0] in IT for dealing with a compromise is, more or less, "kill it with fire". Wipe everything and replace with known good backups. And my opinion of basically anything AI writes is "less trustworthy than random apps you get off the Google Play Store". Likewise, "Airgap it behind a serial console" used to be state of the art for computers that handle root CA key material.
I also wouldn't necessarily call myself an AGI/ASI believer - it's my belief these models are actually still subhuman in capability. But they are also superintelligent in one particular direction: speed. Once you have a model that can do something, it can do that something 100x faster than a person on suitably capable hardware; and we've had the hardware to do that for at least a decade. They can also be ran in parallel. Which means you can throw a lot of bullshit at the wall.
So long as there's some kind of process that lets a particular context learn has already been tried and failed (see what I mentioned above about qntm's There is no Antimemetics Division series), eventually one of these models will break containment. The only thing that changes is how many instances you have to run to get a breach. At some point model capability will catch up to hardware limitations and the explosive growth of AI capability will slow.
I'm not sure if I want to call this the "dumb ASI" theory or the "superheated bucket of water" theory yet.
As for automated attacks, that's already the background radiation of the Internet. There's whole frameworks (e.g. Metasploit) for building automated scanners for known CVEs; I would not be surprised if existing (non-Mythos-class) LLMs are already capable of turning a CVE report into a Metasploit module. The usual cybercrime ecosystem is that someone runs an automated scanner on hijacked machines[1], then they compromise the machines that they find and add it to their botnet. This is done by people with almost no actual programming or security skill copypasting commands from PDFs they bought from an exploit dealer. It's all script kiddies.
What changes with a Mythos-class model is that instead of copypasting commands from PDFs, they can ask the model to find an exploit, and possibly get an exploit chain out of it that nobody has seen before. "NOBUS[2]" vulns used to be the exclusive domain of nation-state actors and zero-day brokers spending millions of dollars on exploit kits; but now all of that is potentially under the domain of randos - at least, until the backlog of obvious vulns the CIA had been stockpiling finally gets cleared out, and the Internet returns to merely being as hazardous to your health as the 2b2t spawn.
[0] Okay, the "hardware SATA overlay" idea is, AFAIK, never been tried before.
[1] I have personally been victimized by this
[2] NObody But US
Re: Responding to the next frontier of critical cyber capabilities
#186There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…
> continued using Artifactory for their sandbox
This is still fine. Infact, they had gone one step ahead by having an internal cluster of artifactory rather public managers like pip. The thing they missed is they didn't revoke the write access to it. Even after the ssrf.
In our[1] or other sandbox providers' sandboxes, by default you have access to npm, pip etc package managers, but only read access.
Re: Responding to the next frontier of critical cyber capabilities
#187I refuse to accept “cyber” as a noun to mean “cybersecurity”. If you use cyber as a noun, I’m assuming you mean this, regardless of context: http://cyber.urbanup.com/521436
Seriously. We already lost "crypto" to such carelessness.
Re: Responding to the next frontier of critical cyber capabilities
#188There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…
[deleted]
Re: Responding to the next frontier of critical cyber capabilities
#189There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…
Anyone else struck by the similarities between the description an "The Cookie Monster” by Vernor Vinge?
Re: Responding to the next frontier of critical cyber capabilities
#190[flagged]