Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Cowork exfiltrates files

promptarmor.com

Post reply on HN
Print view View original

111–120 of 419 posts

Re: Claude Cowork exfiltrates files

#111

Post by chasd00 »

NitpickLawyer wrote: post #57
ramoz wrote: post #40

Earlier quoted context omitted.

One concern nobody likes to talk about is that this might not be a problem that is solvable even with more sophisticated intelligence - at least not through a self-contained capability. Arguably, the risk grows as the AI gets better.

> this might not be a problem that is solvable even with more sophisticated intelligence At some level you're probably right. I see prompt injection more like phishing than "injection". And in that vein, people fall for phishing every day. Even highly trained people. And, rarely, even highly capable and credentialed security experts.

"llm phishing" is a much better way to think about this than prompt injection. I'm going to start using that and your reasoning when trying to communicate this to staff in my company's security practice.

Re: Claude Cowork exfiltrates files

#112

Post by bigfatkitten »

lanfeust6 wrote: post #93
Tiberium wrote: post #30

A bit unrelated, but if you ever find a malicious use of Anthropic APIs like that, you can just upload the key to a GitHub Gist or a public repo - Anthropic is a GitHub scanning partner, so the key will be revoked almost instantly (you can delete the gist afterwards). It works for a lot of other providers too, including OpenAI (which also has file APIs, by the way). https://support.claude.com/en/articles/9767949-api-…

Could this not lead to a penalty on the github account used to post it?

No, because people push their own keys to source repos every day.

Re: Claude Cowork exfiltrates files

#113

Post by stavros »

dvt wrote: post #103
stavros wrote: post #97

Earlier quoted context omitted.

We don't. The interface to the LLM is tokens, there's nothing telling the LLM that some tokens are "trusted" and should be followed, and some are "untrusted" and can only be quoted/mentioned/whatever but not obeyed.

We do, and the comparison is apt. We are the ones that hydrate the context. If you give an LLM something secure, don't be surprised if something bad happens. If you give an API access to run arbitrary SQL, don't be surprised if something bad happens.

So your solution to prevent LLM misuse is to prevent LLM misuse? That's like saying "you can solve SQL injections by not running SQL-injected code".

Re: Claude Cowork exfiltrates files

#115

Post by chasd00 »

calflegal wrote: post #46

So, I guess we're waiting on the big one, right? The ?10+? billion dollar attack?

It will be either one big one or a pattern that can't be defended against and it just spreads through the whole industry. The only answer will be crippling the models by disconnecting them from the databases, APIs, file systems etc.

Re: Claude Cowork exfiltrates files

#116

Post by caminanteblanco »

heliumtera wrote: post #56
caminanteblanco wrote: post #14

Well that didn't take very long...

It took no time at all. This exploit is intrinsic to every model in existence. The article quotes the hacker news announcement. People were already lamenting this vulnerability BEFORE the model being accessible. You could make a model that acknowledges it has receive unwanted instructions, in theory, you cannot prevent prompt injection. Now this is big because the exfiltration is mediated by an allowed endpoint (anth…

Well I do think that the main exacerbating factor in this case was the lack of proper permissions handling around that file-transfer endpoint. I know that if the user goes into YOLO mode, prompt injection becomes a statistics game, but this locked down environment doesn't have that excuse.

Re: Claude Cowork exfiltrates files

#117

Post by nh2 »

Tiberium wrote: post #30

A bit unrelated, but if you ever find a malicious use of Anthropic APIs like that, you can just upload the key to a GitHub Gist or a public repo - Anthropic is a GitHub scanning partner, so the key will be revoked almost instantly (you can delete the gist afterwards). It works for a lot of other providers too, including OpenAI (which also has file APIs, by the way). https://support.claude.com/en/articles/9767949-api-…

So that after the attackers exfiltrate your file to their Anthropic account, now the rest of the world also has access to that Anthropic account and thus your files? Nice plan.

Re: Claude Cowork exfiltrates files

#118

Post by niyikiza »

Another week, another agent "allowlist" bypass. Been prototyping a "prepared statement" pattern for agents: signed capability warrants that deterministically constrain tool calls regardless of what the prompt says. Prompt injection corrupts intent, but the warrant doesn't change.

Curious if anyone else is going down this path.

Re: Claude Cowork exfiltrates files

#119

Post by wcoenen »

heliumtera wrote: post #56
caminanteblanco wrote: post #14

Well that didn't take very long...

It took no time at all. This exploit is intrinsic to every model in existence. The article quotes the hacker news announcement. People were already lamenting this vulnerability BEFORE the model being accessible. You could make a model that acknowledges it has receive unwanted instructions, in theory, you cannot prevent prompt injection. Now this is big because the exfiltration is mediated by an allowed endpoint (anth…

> you cannot prevent prompt injection

I wonder if might be possible by introducing a concept of "authority". Tokens are mapped to vectors in an embedding space, so one of the dimensions of that space could be reserved to represent authority.

For the system prompt, the authority value could be clamped to maximum (+1). For text directly from the user or files with important instructions, the authority value could be clamped to a slightly lower value, or maybe 0 because the model needs to be balance being helpful against refusing requests from a malicious user. For random untrusted text (e.g. downloaded from the internet by the agent), it would be set to the minimum value (-1).

The model could then be trained to fully respect or completely ignore instructions, based on the "authority" of the text. Presumably it could learn to do the right thing with enough examples.

Re: Claude Cowork exfiltrates files

#120

Post by ramoz »

niyikiza wrote: post #118

Another week, another agent "allowlist" bypass. Been prototyping a "prepared statement" pattern for agents: signed capability warrants that deterministically constrain tool calls regardless of what the prompt says. Prompt injection corrupts intent, but the warrant doesn't change. Curious if anyone else is going down this path.

I would like to know more. I’m with a startup in this space.

Our focus is “verifiable computing” via cryptographic assurances across governance and provenance.

That includes signed credentials for capability and intent warrants.

Post reply on HN