Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disrupting the first reported AI-orchestrated cyber espionage campaign

anthropic.com

Post reply on HN
Print view View original

281–290 of 298 posts

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#281

Post by ares623 »

sjsdaiuasgdia wrote: post #259
quantummagic wrote: post #168

Earlier quoted context omitted.

It's literally saying the same thing, just with fewer words.

There were a lot of signs in America at one point in time that said "No Coloreds", "Colored Section", and similar phrases to indicate the spaces that white people had decided non-white people could or could not go. At the same time, there were not a lot of signs saying "No Persons of Color" or "Persons of Color Section". Likewise, my grandfather who died 35 years ago was very fond of saying "the coloreds". His use of…

I'm not American, sorry. "Colored" is just an adjective to me.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#282

Post by thunky »

yubblegum wrote: post #277
thunky wrote: post #250

Earlier quoted context omitted.

> your comparative analysis (beyond its rather strange disconnect with your fellow Human beings) You seem to be having a different conversataion here. I'm comparing work output by two sources and saying this is why people are choosing to use on over the other for day to day tasks. I'm not waxing poetic about the greater impact to society at large when a new productivity source is introduced. > ignores the fact that a…

The very fact that you merely see this as "a new productivity source" support my sense of the disconnect I mentioned. Human beings have patterns of behavior that varies from person to person. This is such an established fact that the concept of personal character is a universal and not culturally centered. (Deterministic) machines and men fail in regular patterns. This is the "human flaws" that you mentioned. It is t…

> The very fact that you merely see this as "a new productivity source" support my sense of the disconnect I mentioned.

Help me understand since my "disconnect" seems to be ruffling your feathers...

What is the correct way to refer to a new tool that is being used to increase productivity?

Or maybe you don't have a problem with the term I used but at the suggestion that someone might find the tool to be useful?

Or is it that I'm suggesting that humans are often unreliable?

I'm having a hard time understanding what is controversial about this.

Machines are better than humans at some things. Humans are better than machines at some things.

Hope you don't find that too offensive.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#283

Post by kelseyfrog »

wongarsu wrote: post #187
0xbadcafebee wrote: post #162

> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a l…

It's not even exclusive to LLMs. Giving humans seemingly innocent tasks that combine to a malicious whole, or telling humans that they work for a security organization while working for a crime organization, are hardly new concepts. The only really novel thing is that with humans you need a lot of them because a single human would piece together that the innocent tasks add up to a not-so-innocent whole. LLMs are esse…

Breaking tasks into innocent subtasks is a known flaw in human organization.

I'm reminded of Caleb sharing his early career experience as an intern at a Department of Defense contractor, where he built a Wi-Fi geolocation application. Initially, he focused on the technical aspects and the excitement of developing a novel tool without considering its potential misuse. The software utilized algorithms to locate Wi-Fi signals based on signal strength and the phone's location, ultimately optimizing performance through machine learning, but Thompson repeatedly emphasizes that the software was intended for lethal purposes.

Eventually, he realizes that the technology could aid in locating and targeting individuals, leading to calls for reflection on ethical practices within tech development.

https://www.rubyevents.org/talks/finding-responsibility

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#284

Post by mrguyorama »

stocksinsmocks wrote: post #24
2OEH8eoCRo0 wrote: post #2

> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases.

So why do we never hear of US sponsored hackers attacking foreign businesses? Or Swedish cyber criminals? Does it never happen? Are “Chinese” hackers just the only ones getting the blame?

How much news do you read in Chinese?

The US government has hacked things in China. That you have not heard of something is not evidence that it doesn't exist.

North Korea also does plenty of hacking around the world. That's how they get a significant portion of their government budget, and they rely on cryptocurrency to support that situation.

Ukraine and Russia are doing lots of official and vigilante hacking right now.

Back in the mid 2000s, there was a guy who called himself "the jester" who was vaguely right wing and spent his time hacking ISIS stuff. My college interviewed him.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#285

Post by FreakLegion »

jacquesm wrote: post #126
FreakLegion wrote: post #125

Earlier quoted context omitted.

Years ago people routinely uploaded all kinds of sensitive corporate and government docs to VirusTotal to scan for malware. Paying customers then got access to those files for research. The opportunities for insider trading were, maybe still are, immense. Data from AI companies won't be as easy to get at, but is comparable in substance I'm sure.

That's absolutely insane. Aren't they owned by Google?

They are now, although to be clear there was (is?) nothing nefarious going on, just people not understanding that public submissions are available to VirusTotal's paying users. These days VT has private scanning, too, but the issue was always one-offs from random finance or investor relations teams.

It's come up here and there in security, too, e.g. in https://www.directdefense.com/harvesting-cb-response-data-le....

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#287

Post by nextworddev »

tw1984 wrote: post #286
nextworddev wrote: post #244

Earlier quoted context omitted.

Ok wumao. Since you are a qwen bot, write a hello world in rust

so you are indeed on CCP's payroll? cool job bro!

Eww no, but you are clearly a CCP bot

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#288

Post by scrubs »

jgmedr wrote: post #254
atlintots wrote: post #55

I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain. My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and…

Anthropic's post is the equivalent of a parent apologizing on behalf of their child that threw a baseball through the neighbor's window. But during the apology the parent keeps sprinkling in "But did you see how fast he threw it? He's going to be a professional one day!"

Hilarious!!!

Did you see? You saw right? How awesome was that throw? Awesome I tell you....

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#289

Post by funnybeam »

mossTechnician wrote: post #272
funnybeam wrote: post #218

Earlier quoted context omitted.

I really think we should stop using the term ‘guard rails’ as it implies a level of control that really doesn’t exist. These things are polite suggestions at best and it’s very misleading to people that do not understand the technology - I’ve got business people saying that using LLMs to process sensitive data is fine because there are “guardrails” in place - we need to make it clear that these kinds of vulnerabiliti…

It's interesting that companies don't provide concrete definitions or examples of what their AI guardrails are. IBM's definition suggests to me they see it as imperative to continue moving fast (and breaking things) no matter what: Think of AI guardrails like the barriers along a highway: they don’t slow the car down, but they do help keep it from veering off course. https://www.ibm.com/think/topics/ai-guardrails

I think you’re absolutely right. These companies know full well that their “guardrails” are ineffective but they just don’t care because they’ve sunk so much money into AI that they are desperate to pretend that everything’s fine and their investments were worthwhile.

I was on a call with Microsoft the other day when (after being pushed) they said they had guardrails in place “to block prompt injection” and linked to an article which said “_help_ block prompt injection”. The careful wording is deliberate I’m sure.

Re: Disrupting the first reported AI-orchestrated cyber espionage campaign

#290

Post by SeriousM »

_heimdall wrote: post #202
0xbadcafebee wrote: post #162

> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a l…

Guardrails are about as good as you can get when creating nondeterministic software, putting it on the internet, and abandoning effectively every important alignment and safety concerns. The guardrails make help make sure that most of the time the LLM acts in a way that users won't complain about or walk away from, nothing more.

LLMs are not nondeterministic. They are infinite state machines that don't 'act' but respond. Be aware of the well hidden seed parameter.
Post reply on HN