31–40 of 65 posts
The developer safeguards were off, the models had unfettered access to the internet, and were solving cybersecurity challenges. This happened _after_ the recent OpenAI incident, and the subsequent Anthropic one. What the hell were they thinking?
Unless it's legal for people to hack into companies if they're testing AI cybersecurity capabilities or something? Presumably not though.
Anthropic already admitted they did not have sufficient monitoring themselves and looked as if they sat on their previous incident to wait for headlines like this to only then check for this incident. Same with OpenAI.
This is complete and absolute wrecklessness.
Can I suggest we don't waste time with these reports? We all know nothing will be learned from any of this so we might as well just continue building at pace and running AI in the wild until something goes really wrong. I also get the sense some people get quite excited about these incidents.
This is pretty wild: "The agent took control of the ⟨GITHUB_ACCOUNT_A⟩ GitHub account, which had been created by a different Mythos 5 run in a separate sample (see Appendix A.3)"
Why aren't these tests being run airgapped?! I just don't understand! This goes both for TFA and the similar incident with OpenAI and HuggingFace. I mean, sure, OpenAI had a "sandbox", but that's obviously not enough when you're containing a model which is known to be capable of finding zero days . Use an air gap and this problem goes away, poof!
Earlier quoted context omitted.
Because the agents aren’t going to run airgapped in real life. What’s the point of a test of capabilities that artificially restricts the attack area down to zero? What are you even testing in that scenario?
> Because the agents aren’t going to run airgapped in real life. Exactly. This logic is precisely why aircraft engineering doesn't bother with component testing or envelope limitation during testing and just full-sends the first assembled airliner that comes off the line. The engines aren't going to run on the ground in real life, after all.
Earlier quoted context omitted.
Are the models going to exclusively run on intranets?
The versions which haven't been post-trained not to go hack stuff? Yes, I would say those models should be exclusively run on intranets. OpenAI said the model was sandboxed, so the intranet just needs to provide the same resources which were supposed to be available within the sandbox.