Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disrupting the first reported AI-orchestrated cyber espionage campaign

anthropic.com

Post reply on HN
Print view View original

41–50 of 298 posts

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#41

Post by tabbott »

Unfortunately, cyber attacks are an application that AI models should excel at. Mistakes that in normal software would be major problems will just have the impact of wasting resources, and it's often not that hard to directly verify whether it in fact succeeded.

Meanwhile, AI coding seems likely to have the impact of more security bugs being introduced in systems.

Maybe there's some story where everyone finds the security bugs with AI tools before the bad guys, but I'm not very optimistic about how this will work out...

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#42

Post by neilv »

It sounds like they directly used Anthropic-hosted compute to do this, and knew that their actions and methods would be exposed to Anthropic?

Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much?

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#43

Post by synapsomorphy »

kenjackson wrote: post #36

Curious why they didn't use DeepSeek... They could've probably built one tuned for this type of campaign.

Chinese builders are not equal to Chinese hackers (even if the hackers are state sponsored). I doubt most companies would be interested in developing hacking tools. Hackers use the best tools available at their disposal, Claude is better than Deepseek. Hacking-tuned LLMs seems like a thing that might pop up in the future, but it takes a lot of resources. Why bother if you can just tell Claude it's doing legitimate work?

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#44

Post by hastamelo »

neilv wrote: post #42

It sounds like they directly used Anthropic-hosted compute to do this, and knew that their actions and methods would be exposed to Anthropic? Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much?

firewalls? anthropic surely is whitelisted.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#45

Post by blibble »

kenjackson wrote: post #35
tantalor wrote: post #26

This feels a lot like aiding & abetting a crime. > Claude identified and tested security vulnerabilities in the target organizations’ systems by researching and writing its own exploit code > use Claude to harvest credentials (usernames and passwords) Are they saying they have no legal exposure here? You created bespoke hacking tools and then deployed them, on your own systems. Are they going to hide behind the old,…

"it's not our fault if you misuse the product to commit a crime that's on you" I feel like if guns can get by with this line then Claude certainly can. Where gun manufacturers can be held liable is if they break the law then that can carry forward. So if Claude broke a law then there might be some additional liability associated with this. But providing a tool seems unlikely to be sufficient to be liable in this case…

if anthropic were selling the product and then had no further control your analogy with guns would be accurate

here they are the ones loading the gun and pulling the trigger

simply because someone asked them to do it nicely

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#46

Post by hastamelo »

Imnimo wrote: post #25

>At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a le…

humans aren't randomly dropped in a random terminal and asked to hack things.

but for models this is their life - doing random things in random terminals

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#47

Post by hastamelo »

tantalor wrote: post #26

This feels a lot like aiding & abetting a crime. > Claude identified and tested security vulnerabilities in the target organizations’ systems by researching and writing its own exploit code > use Claude to harvest credentials (usernames and passwords) Are they saying they have no legal exposure here? You created bespoke hacking tools and then deployed them, on your own systems. Are they going to hide behind the old,…

with your logic linux should have legal exposure because a lot of hackers use linux

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#48

Post by input_sh »

CGMthrowaway wrote: post #22

So basically, Chinese state-backed hackers hijacked Claude Code to run some of the first AI-orchestrated cyber-espionage, using autonomous agents to infiltrate ~30 large tech companies, banks, chemical manufacturers and government agencies. What's amazing is that AI executed most of the attack autonomously, performing at scale and speed unattainable by human teams - thousands of operations per second. A human operato…

What exactly did they hijack? They used it like any other user.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#49

Post by mschwaig »

XorNot wrote: post #38
mschwaig wrote: post #32

I think as AI gets smarter, defenders should start assembling systems how NixOS does it. Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed. Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running sy…

Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long. In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.

Yes, the cycle times are bad and some ecosystems and tasks are a real pain still.

I also agree with you when it comes to the task of auditing every line of Nix code that factors into a given system. Nix doesn't really make things easier there.

The benefit I'm seeing really comes from composition making it easier to share and direct auditing effort.

All of the tricky code that's hard to audit should be relied on and audited by lots of people, while as a result the actual recipe to put together some specific package or service should be easier to audit.

Additionally, I think looking at diffs that represent changes to the system vs reasoning about the effects of changes made through imperative commands that can affect arbitrary parts of the system has similar efficiency gains.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#50

Post by jagged-chisel »

ddalex wrote: post #27
CGMthrowaway wrote: post #22

So basically, Chinese state-backed hackers hijacked Claude Code to run some of the first AI-orchestrated cyber-espionage, using autonomous agents to infiltrate ~30 large tech companies, banks, chemical manufacturers and government agencies. What's amazing is that AI executed most of the attack autonomously, performing at scale and speed unattainable by human teams - thousands of operations per second. A human operato…

how did the autonomous agents inflitrate tech companies ?

Carefully. Expertly. With panache, even.
Post reply on HN