IMO this is the right move.
OpenAI messed up and they are saying they will pause so they can do better.
They are not saying that other orgs who may already be doing better should pause.
Responding to the next frontier of critical cyber capabilities
61–70 of 208 posts
Re: Responding to the next frontier of critical cyber capabilities
#62There's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch. tl;dw; - agents found a way to communicate between several instances during a training run…
Training run was reinforcement learning. It's at 10:10 in the video.
The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.
Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.
I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.
It isn't science, more like a computer game.
Re: Responding to the next frontier of critical cyber capabilities
#63By "cyber" they mean "cybersecurity".
Re: Responding to the next frontier of critical cyber capabilities
#64Earlier quoted context omitted.
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…
I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about. If you consider that incompetence, it’s possible that you’re not a very nice person.
People seem to think LLMs are sentient and not tightly coupled to server infrastructure OAI are in full control over. They literally have every single message sent and action taken, if they can’t monitor that during test runs they should find another job.
It’s like running a benchmark and forgetting to tune it and then wondering why production is running slow.
Re: Responding to the next frontier of critical cyber capabilities
#65I've always felt it's a bit awkward to use terms like 'cyber', 'cyberwarfare' etc it's very Washington D.C. Cybersec would be a better compact term in my book
Or just "security" since the context here is computer and networking stuff.
Re: Responding to the next frontier of critical cyber capabilities
#66Am I the only one not understanding the issue around increased Cybersecurity capabilities? If we consider the amount of RCE/CVE in a software to be limited, I expect these models to result in massively more secured softwares, not less.
If we also consider LLM developed software to have exponential growth, then the CVEs will also grow at an exponential (if proportionally limited) rate. Squash some, create some, repeat. An ever revolving door of vulnerabilities. Will they resolve (and patch and deploy) them faster than they can create them? One can hope.
Re: Responding to the next frontier of critical cyber capabilities
#67Earlier quoted context omitted.
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…
Right? Like I feel like I’m taking crazy pills. OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”
If they did any damage that would be a reasonable argument. As far as I am aware, nothing bad happened.
Re: Responding to the next frontier of critical cyber capabilities
#68Isn't this the opposite of what everyone is saying should happen? That is, lead with open models -- or at least "openness" and don't leave the capabilities in the hands of an elite few? Did they learn nothing from the Hugging Face incident, where HF wasn't even able to use the models to defend itself from OAI's attack?
Re: Responding to the next frontier of critical cyber capabilities
#69Am I the only one not understanding the issue around increased Cybersecurity capabilities? If we consider the amount of RCE/CVE in a software to be limited, I expect these models to result in massively more secured softwares, not less.
Not a security guy but my understanding is: you only need to find one flaw to exploit a system, to make a system totally secure you need to find them all. It's inherently easier to use these tools offensively rather than defensively.
I’m a cybersecurity guy.
>” you only need to find one flaw to exploit a system”
I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture.
Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time.
Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI.
AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.
Re: Responding to the next frontier of critical cyber capabilities
#70Earlier quoted context omitted.
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent. I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently. Edit: not to…
I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about. If you consider that incompetence, it’s possible that you’re not a very nice person.
Also, this is a (semi-intentionally) evolutionary process where any communication medium that was visible to monitoring would disappear.
So by definition the only ones that appear are the ones that are not visible to monitoring.
If:
1. you have something that can find RCE's in leading commercial systems
2. its training gives it drives to communicate successfully with its peers
3. you are a leading commercial system
4. you run it ~10^10 times (the number they gave in the talk)
...it's really hard to have strong certainty up front that it's not going to end up successfully communicating with its peers.