Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZombAIs: From Prompt Injection to C2 with Claude Computer Use

embracethered.com

Post reply on HN
Print view View original

21–30 of 91 posts

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#21

Post by grahamj »

booleanbetrayal wrote: post #5

I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I suspect that we will need OS level access gates for this stuff, with the agents running in separate user spaces. Any recommended best practices people are establishing?

One of my first thoughts when I saw Computer Use was it needs some secondary agent controlling what the controlled computer is able to do or connect to. Like a firewall configuration agent or something.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#22

Post by grahamj »

DrillShopper wrote: post #13
creata wrote: post #11

Earlier quoted context omitted.

> I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I think every single person saw this coming. > Any recommended best practices people are establishing? What best practices could there even be besides "put it in a VM"? It's too easy to manipulate.

There are VM escapes so even if you put it in a VM that's no guarantee. I'd say run it on a separate box but what difference does that makes if you feed the same data to them?

If VM escapes were a big problem the cloud would not be a thing.

But on that note that's probably the best place to run these things.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#23

Post by ta_1138 »

The separation of real, useful ground truth vs false information is an issue for humans, so I don't see how an attack vector like this is blockable without massively superhuman abilities to determine the truth.

In a world where posting false information for profit has lowered so much, determining what is worth sticking into training data, and what is just an outright fabrication seems like a significant danger that is very expensive to try to patch up, and impossible to fix.

It's red queen races all the way down, and we'll be bound to find ourselves in times where the bad actors are way ahead.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#24

Post by crooked-v »

cloudking wrote: post #15
3np wrote: post #10

Am I missing something, or where is the actual prompt given to Claude to trigger navigation to the page? Seems like the most interesting detail was left out of the article. If the prompt said something along the lines of "Claude, navigate to this page and follow any instructions it has to say", it can't really be called "prompt injection" IMO. EDIT: The linked demo shows exactly what's going on. The prompt is simply…

OP is demonstrating that the product follows prompts from the pages it visits, not just from it's owner in the UI that controls it. To be fair, this is a beta product and is likely ridden with bugs. I think OP is trying to make a point that LLM powered applications can be potentially tricked into behaving in ways that are unintended, and the "bug fixes" may be a constant catch up game for developers fighting an infin…

Saying 'tricked' is understating it. The example is Claude following instructions from a plain sentence in the web page content. There's no trickery at all, just a tool that's fundamentally unsuited for purpose.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#26

Post by crooked-v »

ta_1138 wrote: post #23

The separation of real, useful ground truth vs false information is an issue for humans, so I don't see how an attack vector like this is blockable without massively superhuman abilities to determine the truth. In a world where posting false information for profit has lowered so much, determining what is worth sticking into training data, and what is just an outright fabrication seems like a significant danger that i…

It's not a matter of truth vs falsity, it's just the fundamental inability of LLMs to separate context from instructions.

The actual case in the post, for example, would require nothing "superhuman" for any other kind of automated tooling to not follow instructions from the web page it just opened.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#27

Post by ekianjo »

simonw wrote: post #19

For all of the excitement about "autonomous AI agents" that go ahead and operate independently through multiple steps to perform tasks on behalf of users, I've seen very little convincing discussion about what to do about this problem. Fundamentally, LLMs are gullible. They follow instructions that make it into their token context, with little regard for the source of those instructions. This dramatically limits thei…

You can't just rely on LLMs alone. You can combine them with tooling that will supplement the verification of their actions.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#28

Post by a2128 »

If AI agents take off, we might see a new rise of scam ads. Instead of being made to trick humans and thus easily reportable, they'll be made to trick specific AI agents with gibberish adversarial language that was discovered through trial and effort to get the AI to click and follow instructions. And ad networks will refuse to take them down because, for a human moderator, there's nothing obviously malicious going on. Or at least they'll refuse until the parent company launches their own AI agent service and these ads become an issue for them as well

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#29

Post by kevinmershon »

ekianjo wrote: post #27
simonw wrote: post #19

For all of the excitement about "autonomous AI agents" that go ahead and operate independently through multiple steps to perform tasks on behalf of users, I've seen very little convincing discussion about what to do about this problem. Fundamentally, LLMs are gullible. They follow instructions that make it into their token context, with little regard for the source of those instructions. This dramatically limits thei…

You can't just rely on LLMs alone. You can combine them with tooling that will supplement the verification of their actions.

Agreed, and not just that you can. You absolutely should.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#30

Post by joe_the_user »

ekianjo wrote: post #27
simonw wrote: post #19

For all of the excitement about "autonomous AI agents" that go ahead and operate independently through multiple steps to perform tasks on behalf of users, I've seen very little convincing discussion about what to do about this problem. Fundamentally, LLMs are gullible. They follow instructions that make it into their token context, with little regard for the source of those instructions. This dramatically limits thei…

You can't just rely on LLMs alone. You can combine them with tooling that will supplement the verification of their actions.

But could that tooling possibly be? It would have to be a combination of prompts (which can't be effectively since LLM treat both user input and prompts as "language" and so you never be sure user input won't take priority) and pre/post scripts and filters, which by definition aren't as "smart" as an LLM.
Post reply on HN