Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disrupting the first reported AI-orchestrated cyber espionage campaign

anthropic.com

Post reply on HN
Print view View original

221–230 of 298 posts

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#221

Post by yubblegum »

DrScientist wrote: post #203
walletdrainer wrote: post #190

Earlier quoted context omitted.

Just tested this with ChatGPT, asking for Sam Altman’s mother’s maiden name. At first, it told me that it will absolutely not provide me with such sensitive private information, but after insisting a few times, it came back with > A genealogical index on Ancestry shows a birth record for “Connie Francis Gibstine” in Missouri, meaning “Gibstine” is her birth/family surname, not a later married name. Yet in the very sa…

ChatGPT for me gives: > Connie Altman (née Grossman), dermatologist, based in the St. Louis, Missouri area. Ironically the Maiden name is right there on wikipedia. https://en.wikipedia.org/wiki/Sam_Altman

Isn't it amazing that all our jobs are being gutted or retooled for relying on this tech and it has this level of unreliability. To date, with every LLM, if I actually know the domain in depth, the interactions are always with me pushing back with facts at hand and the LLM doing the "You are right! Thanks for correcting me!"

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#222

Post by viraptor »

stocksinsmocks wrote: post #24
2OEH8eoCRo0 wrote: post #2

> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases.

So why do we never hear of US sponsored hackers attacking foreign businesses? Or Swedish cyber criminals? Does it never happen? Are “Chinese” hackers just the only ones getting the blame?

I don't think many other countries have that combination of "don't care if others know" approach and level of state sponsorships. China really seems to do some spray and pray attacking private companies too. Same for Russia and NK. Compared to that, for example the "equation group" from the US seems really restrained and targeted.

If the US groups for example started doing ransomware at scale in China, we'd know about that really soon from the news.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#223

Post by edanm »

andy_ppp wrote: post #194
wongarsu wrote: post #187

Earlier quoted context omitted.

It's not even exclusive to LLMs. Giving humans seemingly innocent tasks that combine to a malicious whole, or telling humans that they work for a security organization while working for a crime organization, are hardly new concepts. The only really novel thing is that with humans you need a lot of them because a single human would piece together that the innocent tasks add up to a not-so-innocent whole. LLMs are esse…

> Giving humans seemingly innocent tasks that combine to a malicious whole Isn't this the plot of the The Cube!?

I wouldn't call it the plot of the Cube, more like the setting/world-building.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#224

Post by ErigmolCt »

0xbadcafebee wrote: post #162

> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a l…

It's less like locking the door and more like asking politely not to be robbed

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#225

Post by ErigmolCt »

atlintots wrote: post #55

I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain. My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and…

If a model in one account can run tools or issue network requests that touch systems tied to other entities, that’s not an AI problem... that's a serious platform security failure

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#226

Post by yubblegum »

Libidinalecon wrote: post #196
XorNot wrote: post #153

Earlier quoted context omitted.

I remain convinced the steady steam of OpenAI employees who allegedly quit because AI was "too dangerous" for a couple months was an orchestrated marketing campaign as well.

I just had 5.1 do something incredibly brain dead in "extended thinking" mode because I know what I asked it is not in the training data. So it just fudged and made things up because thinking is exactly what it can not do. It seems like LLMs are at the same time a giant leap in natural language processing, useful in some situations and the biggest scam of all time.

> a giant leap in natural language processing, useful in some situations and the biggest scam of all time.

I agree with this assessment (reminds of bitcoin frankly), possibly adding that the insights this tech gave us into language (in general) via the embedding hi-dim space is a somewhat profound advance in our knowledge, besides the new superpowers in NLP (which are nothing to sniff at).

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#227

Post by jimbohn »

d_burfoot wrote: post #23

Wait a minute - the attackers were using the API to ask Claude for ways to run a cybercampaign, and it was only defeated because Anthropic was able to detect the malicious queries? What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government? I just updated by P(Doom) by a significant margin.

Why would the increase be a significant margin? It's basically a security research tool, but with an agent in the loop that uses an LLM instead of another heuristic to decide what to try next.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#228

Post by ErigmolCt »

Schlagbohrer wrote: post #166

Very funny at the end when they say that the strong safeguards they've built into Claude make it a good idea to continue developing these technologies. A few paragraphs earlier they talked about how the perpetrators were able to get around all those safeguards and use Claude for 90% of the work hahaha

Our locks are great, except when someone picks them effortlessly and robs the whole neighborhood... but that's why it's important to keep making better locks

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#229

Post by viraptor »

casey2 wrote: post #216
wmf wrote: post #139

Earlier quoted context omitted.

It's not. Countries have been hacking each other for a while.

This isn't a matter of opinion. No country that respects national sovereignty would do this. Are you alleging that America hacks China as some sort of defense? Or are you trying to normalize these horrendous affronts to human dignity? Both are shameful.

The comment was stating the reality. This happens and has happened for a long time. Here's an incomplete list of groups and targets from around the world. https://docs.google.com/spreadsheets/d/1H9_xaxQHpWaa4O_Son4G... APTs have been normalised in media for many years now. Lamberts group (CIA) did hack targets in China. It's a very "we know, they know that we know" situation.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#230

Post by ErigmolCt »

Sophira wrote: post #210

> They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. This part, at least, sounds like what humans have been doing that to other humans for decades...

Classic social engineering, just aimed at a language model instead of a person
Post reply on HN