Anyone using Claude for processing sensitive information should be wondering how often it ends up in front of a humans eyes as a false positive
Disrupting the first reported AI-orchestrated cyber espionage campaign
91–100 of 298 posts
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#92Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#93They're spinning this as a positive learning experience, and trying to make themselves look good. But, make no mistake, this was a failure on Anthropic's part to prevent this kind of abuse from being possible through their systems in the first place. They shouldn't be earning any dap from this.
Meh, drama aside, I'm actually curious what would be the true capabilities of a system that doesn't go through any "safety" alignment at all. Like an all out "mil-spec" agent. Feed it everything, RL it to own boxes, and let it loose in an air-gapped network to see what the true capabilities are. We know alignment hurts model performance (oAI people have said it, MS people have said it). We also know that companies tr…
Nous claims to be doing that but I haven't seen much discussion of it.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#94Earlier quoted context omitted.
if anthropic were selling the product and then had no further control your analogy with guns would be accurate here they are the ones loading the gun and pulling the trigger simply because someone asked them to do it nicely
You...do realize Claude is not just a guy sitting in Anthropic's office doing what people on the internet tell him to, right?
That's a good analogy actually.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#95Earlier quoted context omitted.
We should assume sophisticated attackers, AI-enabled or otherwise, as our time with computers goes on, and no longer give leeway to organizations who are unable to secure their systems properly or keep customers safe in the event that they are breached. Decades of warnings from the infosec community have fallen upon the deaf ears of "it doesn't hurt so I'm not going to fix it" of those whose opinions have mattered in…
Would you say the same about all people being responsible for safeguarding their own reputations against reputational attacks at scale, all communities have to protect against advanced persistent threats infiltrating them 24/7, and all people’s immune systems have to protect against designer pathogens by AI-assisted terrorists?
I think our full understanding of the spectrum of these threats will lead to the construction of robust safeguards against them. Reputational attacks at scale are a weakness of the current platforms within which we consume news, form community, and build trust. Computer attacks described in the article are caused by sloppy design/implementation brought into existence by folks whose daily incentives are less about making safe code and more about delivering features. "Designer pathogens" have been described as an accessible form of terrorism since far before AI has existed. All of these threats and similar have existed since before AI, and will continue to exist if AI is snapped out of existence right now. The excuse for not preventing/addressing them has always been about knowledge and development resources, which current generative AI tech addresses.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#96> we detected a highly sophisticated cyber espionage operation
conducted by a Chinese state-sponsored group we've designated GTG-1002
How about calling them something like xXxDragonSlayer69xXx instead? GTG-1002 is almost respectable a name. But xXxDragonSlayer69xXx? is hate to be named that.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#97Earlier quoted context omitted.
I don’t think these agents are doing anything a dedicated human couldn’t do, only enabling it at scale. Relying on “not being one of few they focus on” as security is just security as obscurity. You were living on borrowed time anyway.
An, there it is. The stock reply that comes no matter what the criticism of AI is. I am talking about the international community coming together put COMPETITION aside and start COOPERATING on controlling proliferation of models for malicious AI agents the way the international community SUCCESSFULLY did with chemical weapons and CFCs.
It's one thing for, eg, OpenAI to decide a model is too dangerous to release. I don't really care, they don't owe anyone anything. It's more that open source is going to catch up, and it's a slippery slope into legal regulation that stifles innovation, competition, and won't meaningfully stop hackers from getting these models.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#98Wait a minute - the attackers were using the API to ask Claude for ways to run a cybercampaign, and it was only defeated because Anthropic was able to detect the malicious queries? What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government? I just updated by P(Doom) by a significant margin.
If plain open-source local models were able to do what Claude API does, Anthropic would be out of business. Local models are a different thing than those cloud-based assistants and APIs.
> If plain open-source local models were able to do what Claude API does, Anthropic would be out of business.
Not necessarily. Oracle has made billions selling a database that's less good than plain open-source ones, for example.
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#99> The attackers used AI ... to execute the cyberattacks
Translation: "The attacker's paid us to use our product to execute the cyberattacks"
Re: Disrupting the first reported AI-orchestrated cyber espionage campaign
#100[deleted]