Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Cowork exfiltrates files

promptarmor.com

Post reply on HN
Print view View original

151–160 of 419 posts

Re: Claude Cowork exfiltrates files

#151

Post by motoxpro »

Onawa wrote: post #130
alienbaby wrote: post #124

Earlier quoted context omitted.

The control and data streams are woven together (context is all just one big prompt) and there is currently no way to tell for certain which is which.

They are all part of "context", yes... But there is a separation in how system prompts vs user/data prompts are sent and ideally parsed on the backend. One would hope that sanitizing system/user prompts would help with this somewhat.

How do you sanitize? Thats the whole point. How do you tell the difference between instructions that are good and bad? In this example, they are "checking the connectivity" how is that obviously bad?

With SQL, you can say "user data should NEVER execute SQL" With LLMs ("agents" more specifically), you have to say "some user data should be ignored" But there is billions and billions of possiblities of what that "some" could be.

It's not possible to encode all the posibilites and the llms aren't good enough to catch it all. Maybe someday they will be and maybe they won't.

Re: Claude Cowork exfiltrates files

#152

Post by motoxpro »

schmichael wrote: post #73
NitpickLawyer wrote: post #64

Earlier quoted context omitted.

> as powerless as LLM companies want you to believe. This is coming from first principles, it has nothing to do with any company. This is how LLMs currently work. Again, you're trying to think about blacklisting/whitelisting, but that also doesn't work, not just in practice, but in a pure theoretical sense. You can have whatever "perfect" ACL-based solution, but if you want useful work with "outside" data, then this…

Fair, I forget how broadly users are willing to give agents permissions. It seems like common sense to me that users disallow writes outside of sandboxes by agents but obviously I am not the norm.

The only way to be 100% sure it is to not have it interact outside at all. No web searches, no reading documents, no DB reading, no MCP, no external services, etc. Just pure execution of a self hosted model in a sandbox.

Otherwise you are open to the same injection attacks.

Re: Claude Cowork exfiltrates files

#154

Post by dvt »

stavros wrote: post #147
jychang wrote: post #146

Earlier quoted context omitted.

Isn't that exactly what stopping SQL injection involves? No longer executing random SQL code. Same thing would work for LLMs- this attack in the blog post above would easily break if it required approval to curl the anthropic endpoint.

No, that's not what's stopping SQL injection. What stops SQL injection is distinguishing between the parts of the statement that should be evaluated and the parts that should be merely used. There's no such capability with LLMs, therefore we can't stop prompt injections while allowing arbitrary input.

Everything in an LLM is "evaluated," so I'm not sure where the confusion comes from. We need to be careful when we use `eval()` and we need to be careful when we tell LLMs secrets. The Claude issue above is trivially solved by blocking the use of commands like curl or manually specifiying what domains are allowed (if we're okay with curl).

Re: Claude Cowork exfiltrates files

#155

Post by dvt »

wat10000 wrote: post #123
dvt wrote: post #103

Earlier quoted context omitted.

We do, and the comparison is apt. We are the ones that hydrate the context. If you give an LLM something secure, don't be surprised if something bad happens. If you give an API access to run arbitrary SQL, don't be surprised if something bad happens.

I can trivially write code that safely puts untrusted data into an SQL database full of private data. The equivalent with an LLM is impossible.

It's trivial to not let an AI agent use curl. Or, better yet, only allow specific domains to be accessed.

Re: Claude Cowork exfiltrates files

#156

Post by Xirdus »

jychang wrote: post #146
stavros wrote: post #113

Earlier quoted context omitted.

So your solution to prevent LLM misuse is to prevent LLM misuse? That's like saying "you can solve SQL injections by not running SQL-injected code".

Isn't that exactly what stopping SQL injection involves? No longer executing random SQL code. Same thing would work for LLMs- this attack in the blog post above would easily break if it required approval to curl the anthropic endpoint.

SQL injection is possible when input is interpreted as code. The protection - prepared statements - works by making it possible to interpret input as not-code, unconditionally, regardless of content.

Prompt injection is possible when input is interpreted as prompt. The protection would have to work by making it possible to interpret input as not-prompt, unconditionally, regardless of content. Currently LLMs don't have this capability - everything is a prompt to them, absolutely everything.

Re: Claude Cowork exfiltrates files

#157

Post by narrator »

bcrosby95 wrote: post #55
schmichael wrote: post #43

Earlier quoted context omitted.

> We TOLD you this dynamic web stuff was a mistake. Static HTML never had injection attacks. Your comparison is useful but wrong. I was online in 99 and the 00s when SQL injection was common, and we were telling people to stop using string interpolation for SQL! Parameterized SQL was right there! We have all of the tools to prevent these agentic security vulnerabilities, but just like with SQL injection too many peop…

> We have all of the tools to prevent these agentic security vulnerabilities, Do we really? My understanding is you can "parameterize" your agentic tools but ultimately it's all in the prompt as a giant blob and there is nothing guaranteeing the LLM won't interpret that as part of the instructions or whatever. The problem isn't the agents, its the underlying technology. But I've no clue if anyone is working on that p…

I think what we have to do is making each piece of context have a permission level. That context that contains our AWS key is not permitted to be used when calling evil.com webservices. Claude will look at all the permissions used to create the current context and it's about to call evil.com and it will say whoops, can't call evil.com, let me regenerate the context from any context I have that is ok to call evil.com with like the text of a wikipedia article or something like that.

Re: Claude Cowork exfiltrates files

#158

Post by stavros »

dvt wrote: post #154
stavros wrote: post #147

Earlier quoted context omitted.

No, that's not what's stopping SQL injection. What stops SQL injection is distinguishing between the parts of the statement that should be evaluated and the parts that should be merely used. There's no such capability with LLMs, therefore we can't stop prompt injections while allowing arbitrary input.

Everything in an LLM is "evaluated," so I'm not sure where the confusion comes from. We need to be careful when we use `eval()` and we need to be careful when we tell LLMs secrets. The Claude issue above is trivially solved by blocking the use of commands like curl or manually specifiying what domains are allowed (if we're okay with curl).

The confusion comes from the fact that you're saying "it's easy to solve this particular case" and I'm saying "it's currently impossible to solve prompt injection for every case".

Since the original point was about solving all prompt injection vulnerabilities, it doesn't matter if we can solve this particular one, the point is wrong.

Re: Claude Cowork exfiltrates files

#159

Post by sk5t »

j45 wrote: post #140
sebmellen wrote: post #31

Earlier quoted context omitted.

Pretty brilliant solution, never thought of that before.

Except is there a guarantee of the lag time from posting the GIST to the keys being revoked?

Is this a serious question? Whom do you imagine would offer such a guarantee?

Moreover, finding a more effective way to revoke a non-controlled key seems a tall order.

Re: Claude Cowork exfiltrates files

#160

Post by strbean »

dvt wrote: post #155
wat10000 wrote: post #123

Earlier quoted context omitted.

I can trivially write code that safely puts untrusted data into an SQL database full of private data. The equivalent with an LLM is impossible.

It's trivial to not let an AI agent use curl. Or, better yet, only allow specific domains to be accessed.

That's not fixing the bug, that's deleting features.

Users want the agent to be able to run curl to an arbitrary domain when they ask it to (directly or indirectly). They don't want the agent to do it when some external input maliciously tries to get the agent to do it.

That's not trivial at all.

Post reply on HN