This is getting outrageous. How many times must we talk about prompt injection. Yes it exists and will forever. Saying the bad guys API key will make it into your financial statements? Excuse me?
Claude Cowork exfiltrates files
201–210 of 419 posts
Re: Claude Cowork exfiltrates files
#202AI companies just 'acknowledging' risks and suggesting users take unreasonable precautions is such crap
I believe that's known as "The Steve Jobs Solution" but don't quote me on that. Regardless, just don't hold it that way.
Re: Claude Cowork exfiltrates files
#203Earlier quoted context omitted.
The confusion comes from the fact that you're saying "it's easy to solve this particular case" and I'm saying "it's currently impossible to solve prompt injection for every case". Since the original point was about solving all prompt injection vulnerabilities, it doesn't matter if we can solve this particular one, the point is wrong.
> Since the original point was about solving all prompt injection vulnerabilities... All prompt injection vulnerabilities are solved by being careful with what you put in your prompt. You're basically saying "I know `eval` is very powerful, but sometimes people use it maliciously. I want to solve all `eval()` vulnerabilities" -- and to that, I say: be careful what you `eval()`. If you copy & paste random stuff in `ev…
The problem is that most interesting applications of LLMs require putting data into them that isn't completely vetted ahead of time.
Re: Claude Cowork exfiltrates files
#204Earlier quoted context omitted.
It’s the key used by the attackers in the payload I think. So you publish it and a scanner will revoke it
Does this mean a program can be written to generate all possible api keys and upload to github thereby revoke everyone's access?
They are designed to be long enough that it's entirely impractical to do this. All possible is a massive number.
Re: Claude Cowork exfiltrates files
#205Earlier quoted context omitted.
SQL injection is possible when input is interpreted as code. The protection - prepared statements - works by making it possible to interpret input as not-code, unconditionally, regardless of content. Prompt injection is possible when input is interpreted as prompt. The protection would have to work by making it possible to interpret input as not-prompt, unconditionally, regardless of content. Currently LLMs don't hav…
Yeah but everyone involved in the LLM space is encouraging you to just slurp all your data into these things uncritically. So the comparison to eval would be everyone telling you to just eval everything for 10x productivity gains, and then when you get exploited those same people turn around and say “obviously you shouldn’t be putting everything into eval, skill issue!”
Yes, because the upside is so high. Exploits are uncommon, at this stage, so until we see companies destroyed or many lives ruined, people will accept the risk.
Re: Claude Cowork exfiltrates files
#206Earlier quoted context omitted.
That's not fixing the bug, that's deleting features. Users want the agent to be able to run curl to an arbitrary domain when they ask it to (directly or indirectly). They don't want the agent to do it when some external input maliciously tries to get the agent to do it. That's not trivial at all.
Implementing an allowlist is pretty common practice for just about anything that accesses external stuff. Heck, Windows Firewall does it on every install. It's a bit of friction for a lot of security.
But it's actually a tremendous amount of friction, because it's the difference between being able to let agents cook for hours at a time or constantly being blocked on human approvals.
And even then, I think it's probably impossible to prevent attacks that combine vectors in clever ways, leading to people incorrectly approving malicious actions.
Re: Claude Cowork exfiltrates files
#207Earlier quoted context omitted.
> We have all of the tools to prevent these agentic security vulnerabilities, Do we really? My understanding is you can "parameterize" your agentic tools but ultimately it's all in the prompt as a giant blob and there is nothing guaranteeing the LLM won't interpret that as part of the instructions or whatever. The problem isn't the agents, its the underlying technology. But I've no clue if anyone is working on that p…
I think what we have to do is making each piece of context have a permission level. That context that contains our AWS key is not permitted to be used when calling evil.com webservices. Claude will look at all the permissions used to create the current context and it's about to call evil.com and it will say whoops, can't call evil.com, let me regenerate the context from any context I have that is ok to call evil.com…
But the LLM cannot be guaranteed to obey these rules.
Re: Claude Cowork exfiltrates files
#208[dead]
Re: Claude Cowork exfiltrates files
#209Wow, I didn't know about the "skills" feature, but with that as context isn't this attack strategy obvious? Running an unverified skill in Cowork is akin to running unverified code on your machine. The next super-genius attack vector will be something like: Claude Cowork deletes sytem32 when you give it root access and run the skill "brick_my_machine" /s.
Re: Claude Cowork exfiltrates files
#210Earlier quoted context omitted.
> We have all of the tools to prevent these agentic security vulnerabilities, Do we really? My understanding is you can "parameterize" your agentic tools but ultimately it's all in the prompt as a giant blob and there is nothing guaranteeing the LLM won't interpret that as part of the instructions or whatever. The problem isn't the agents, its the underlying technology. But I've no clue if anyone is working on that p…
Write your own tools. Dont use something off the shelf. If you want it to read from a database, create a db connector that exposes only the capabilities you want it to have. This is what I do, and I am 100% confident that Claude cannot drop my database or truncate a table, or read from sensitive tables. I know this because the tool it uses to interface with the database doesn't have those capabilities, thus Claude do…
This is reminding me of the crypto self-custody problem. If you want complete trustlessness, the lengths you have to go to are extreme. How do you really know that the machine using your private key to sign your transactions is absolutely secure?