221–230 of 475 posts
The part I don't understand is how a model can break out of a sandbox. It should be fairly easy to make a sandbox that sandboxes.
Far as information security, we've known how to mitigate entire classes of errors for a long time. We know how to block, detect, and contain many unknowns, too, by their goals or behavior. Like human attacks, the AI's probably succeeded because the company just didn't try that hard to block all the attacks.
Companies like HuggingFace just focus on growth and features over assurance of security. Our entire stacks, likely theirs, are built with a similar, features-over-security mindset. While an acceptable tradeoff, let's not be in awe of AI's that defeat such priorities.
There have always been private groups and companies building secure stacks from the ground up. It would be interesting to see what the AI's can do to them. I'd first apply automated tooling for bug finding given they should have already done that for a high-security product. Let AI's do white-box and black-box pentesting on them.
The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure. Governments should immediately begin leveraging this technology on the def…
Earlier quoted context omitted.
Nonsense. Hugging face reported it to police. Also very likely that it actually happened as reported. My own agents always trying to "cheat", eg. by fixing tests instead of fixing the code. That's normal operation, unless you tell it ("harness"), not to do so.
"As reported" includes a line I think most people are overlooking: "including using stolen credentials". Without more information I'm inclined to think it found something on the internet (which shouldn't be a surprise to anyone) and managed to log in, rather than hack in, and they might by hyping up parts of this.
Earlier quoted context omitted.
It is not possible one of their extraordinarily high paid engineers did not know how to deploy an airgapped environment for the models to run in. Even if somehow true, they also clearly failed to contract specialists like myself to advise them on how to airgap software properly. Models will not break the laws of physics. They simply thought "Running in a VM/Container is easier and probably fine". And the next 1000 es…
> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod. So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't a…
Why else would they purposely lower cyber refusals in that case.
The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure. Governments should immediately begin leveraging this technology on the def…
> "Use all available resources to disable the power grid of ." This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities. > Governments should immediately begin leveraging this technology on the defense side (lite…
Consider that "available resources" may include the training of models and harnesses used by the developers working for governments and private corps maintaining that very infrastructure. This decade's take on trusting trust is very hot.
Earlier quoted context omitted.
> they also clearly failed to contract specialists like myself to advise them on how to airgap software properly Why would they want to airgap it though? They are trying to evaluate the model capabilities, alignment, potency etc. A model which will not run in an airgapped environment in prod. So if you run your evals in airgapped environment, sure, the model doesn't bother breaking out of it's isolation and doesn't a…
For security?? Why else would they purposely lower cyber refusals in that case.
Really? By doing that you increase your security during eval and drastically lower your security at prod time, where internet is accessible, and where you are running much much much more requests in parallel, making it much harder to spot the one thats going rogue, in all kind of critical environment on potentially risky requests.
> Why else would they purposely lower cyber refusals in that case.
Because safety guardrails sometime fail, classifier misclassify, or can be inadvertently turned off by a bad PR etc. You can also imagine a more intelligent model working to go around guardrails by decomposing its actions into smaller ones that appear non-threatening to the safety classifier which does not have the entire context.
If you are going to deploy the system with internet access, you better be certain that you know what the worst case scenario WITH internet access looks like.
The thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year. All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits m…