171–180 of 298 posts
>At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a le…
Earlier quoted context omitted.
reminds me of the YouTube ads I get that are like "Warning: don't do this new weight loss trick unless you have to lose over 50 pounds, you will end up losing too much weight!". As if it's so effective it's dangerous.
I remain convinced the steady steam of OpenAI employees who allegedly quit because AI was "too dangerous" for a couple months was an orchestrated marketing campaign as well.
Then a quiet conversation, where if things are said about AI, a massive compensation package instead of normal one. Maybe including it as stock.
Along with an NDA.
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a l…
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a l…
The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated Not surprised at all if this is true, but how can they be sure? Access log? They have extraordinary security team? Or some help from three letter agencies?
I've been using Claude to scan my codebase and submit issues and PRs when it finds a potential vulnerability and honestly it's pretty good.
So preventing it from doing any sort of work that can surface vulnerabilities would affect me as a user.
But yeah I'm not sure what the answer is here? Is part of it for the defender to actively use these systems to test itself before going to prod?
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a l…
This is exactly why I make a huge exception for AI models, when it comes to open source software. I've been a big advocate of open source, spending over $1M to build massive code bases with my team, and giving them away to the public. But this is different. AI agents in the wrong hands are dangerous. The reason these guys were even able to detect this activity, analyze it, ban accounts, etc., is because the models ar…
Real advocates of open source software long advocated for running software on their own hardware.
And real real advocates of open source software also advocated for publishing the training data of AI models.