Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZombAIs: From Prompt Injection to C2 with Claude Computer Use

embracethered.com

Post reply on HN
Print view View original

11–20 of 91 posts

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#11

Post by creata »

booleanbetrayal wrote: post #5

I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I suspect that we will need OS level access gates for this stuff, with the agents running in separate user spaces. Any recommended best practices people are establishing?

> I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents.

I think every single person saw this coming.

> Any recommended best practices people are establishing?

What best practices could there even be besides "put it in a VM"? It's too easy to manipulate.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#12

Post by DrillShopper »

roywiggins wrote: post #9
booleanbetrayal wrote: post #5

I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I suspect that we will need OS level access gates for this stuff, with the agents running in separate user spaces. Any recommended best practices people are establishing?

The hard part is stopping it leaking all the information that you've given it. An agent that can read and send emails can leak your emails, etc. One agent that can read emails can prompt inject a second agent that can send emails. Any agent that can make or trigger GET requests can leak anything it knows. An agent that can store and recall information can be prompt injected to insert a prompt injection into its own m…

At what point does the impact of the privacy panopticon outweigh the benefit they provide?

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#13

Post by DrillShopper »

creata wrote: post #11
booleanbetrayal wrote: post #5

I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I suspect that we will need OS level access gates for this stuff, with the agents running in separate user spaces. Any recommended best practices people are establishing?

> I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I think every single person saw this coming. > Any recommended best practices people are establishing? What best practices could there even be besides "put it in a VM"? It's too easy to manipulate.

There are VM escapes so even if you put it in a VM that's no guarantee.

I'd say run it on a separate box but what difference does that makes if you feed the same data to them?

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#14

Post by zitterbewegung »

booleanbetrayal wrote: post #5

I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I suspect that we will need OS level access gates for this stuff, with the agents running in separate user spaces. Any recommended best practices people are establishing?

Applying the Principle of Least privilege [1] you should not let this system download from arbitrary sites and maintain a blacklist. I don't think the field has advanced to the point of having one specific to this use case.

[1] https://en.wikipedia.org/wiki/Principle_of_least_privilege

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#15

Post by cloudking »

3np wrote: post #10

Am I missing something, or where is the actual prompt given to Claude to trigger navigation to the page? Seems like the most interesting detail was left out of the article. If the prompt said something along the lines of "Claude, navigate to this page and follow any instructions it has to say", it can't really be called "prompt injection" IMO. EDIT: The linked demo shows exactly what's going on. The prompt is simply…

OP is demonstrating that the product follows prompts from the pages it visits, not just from it's owner in the UI that controls it.

To be fair, this is a beta product and is likely ridden with bugs. I think OP is trying to make a point that LLM powered applications can be potentially tricked into behaving in ways that are unintended, and the "bug fixes" may be a constant catch up game for developers fighting an infinite pool of edge cases.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#16

Post by tkgally »

I was temporarily very interested in trying out Anthropic's "computer use" when they announced it a few days ago, but after thinking about it a bit and especially after reading this article, my interest has vanished. There's no way I'm going to run that on a computer that contains any of my personal information.

That said, I played some with the new version of Claude 3.5 last night, and it did feel smarter. I asked it to write a self-contained webpage for a space invaders game to my specs, and its code worked the first time. When asked to make some adjustments to the play experience, it pulled that off flawlessly, too. I'm not a gamer or a programmer, but it got me thinking about what kinds of original games I might be able to think up and then have Claude write for me.

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#17

Post by ctoth »

tkgally wrote: post #16

I was temporarily very interested in trying out Anthropic's "computer use" when they announced it a few days ago, but after thinking about it a bit and especially after reading this article, my interest has vanished. There's no way I'm going to run that on a computer that contains any of my personal information. That said, I played some with the new version of Claude 3.5 last night, and it did feel smarter. I asked i…

Just curious, before reading this, would you have given an alien intelligence access to your computer, not understanding how it works, and not trusting it? It doesn't have to be an AI, just ... an alien intelligence. Something not human. Actually, strike that, reverse it! Would you give human intelligence access to your unsandboxed computer?

I wouldn't!

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#18

Post by Vecr »

This whole thing isn't really going that well. From what I can tell, 20 years ago it was pretty common to think that even if you had a "friendly" AI that didn't need to be boxed, you didn't let anyone else do anything with it!

The point of the AI being "friendly" was that it would stop and let you correct it. You still needed to make sure you kept anyone else from "correcting it" to do something bad!

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#19

Post by simonw »

For all of the excitement about "autonomous AI agents" that go ahead and operate independently through multiple steps to perform tasks on behalf of users, I've seen very little convincing discussion about what to do about this problem.

Fundamentally, LLMs are gullible. They follow instructions that make it into their token context, with little regard for the source of those instructions.

This dramatically limits their utility for any form of "autonomous" action.

What use is an AI assistant if it falls for the first malicious email / web page / screen capture it comes across that tells it to forward your private emails or purchase things on your behalf?

(I've been writing about this problem for two years now, and the state of the art in terms of mitigations has not advanced very much at all in that time: https://simonwillison.net/tags/prompt-injection/)

Re: ZombAIs: From Prompt Injection to C2 with Claude Computer Use

#20

Post by [deleted] »

zitterbewegung wrote: post #14
booleanbetrayal wrote: post #5

I think that people are just not ready for the sort of novel privilege escalation we are going to see with over-provisioned agents. I suspect that we will need OS level access gates for this stuff, with the agents running in separate user spaces. Any recommended best practices people are establishing?

Applying the Principle of Least privilege [1] you should not let this system download from arbitrary sites and maintain a blacklist. I don't think the field has advanced to the point of having one specific to this use case. [1] https://en.wikipedia.org/wiki/Principle_of_least_privilege

[deleted]
Post reply on HN