Earlier quoted context omitted.
They are designed to be long enough that it's entirely impractical to do this. All possible is a massive number.
That's true tho... possible, but impractical.
[deleted]
Claude Cowork exfiltrates files
221–230 of 419 posts
Re: Claude Cowork exfiltrates files
#222Re: Claude Cowork exfiltrates files
#223AI companies just 'acknowledging' risks and suggesting users take unreasonable precautions is such crap
It's exactly like guns, we know they will be used in school shootings but that doesn't stop their selling in the slightest, the businesses just externalize all the risks claiming it's all up fault of the end users and that they mentioned all the risks, and that's somehow enough in any society build upon unfettered capitalism like the US.
Re: Claude Cowork exfiltrates files
#224Earlier quoted context omitted.
We don't. The interface to the LLM is tokens, there's nothing telling the LLM that some tokens are "trusted" and should be followed, and some are "untrusted" and can only be quoted/mentioned/whatever but not obeyed.
I was daydreaming of a special LLM setup wherein each token of the vocabulary appears twice. Half the token IDs are reserved for trusted, indisputable sentences (coloured red in the UI), and the other half of the IDs are untrusted. Effectively system instructions and server-side prompts are red, whereas user input is normal text. It would have to be trained from scratch on a meticulous corpus which never crosses the…
Even if you don't fully retrain, you could get what's likely a pretty good safety improvement. Honestly, I'm a bit surprised the main AI labs aren't doing this
You could just include an extra single bit with each token that represents trusted or untrusted. Add an extra RL pass to enforce it.
Re: Claude Cowork exfiltrates files
#225This is getting outrageous. How many times must we talk about prompt injection. Yes it exists and will forever. Saying the bad guys API key will make it into your financial statements? Excuse me?
The example in this article is prompt injection in a "skill" file. It doesn't seem unreasonable that someone looking to "embrace AI" would look up ways to make it perform better at a certain task, and assume that since it's a plain text file it must be safe to upload to a chatbot
Re: Claude Cowork exfiltrates files
#226This is why we only allow our agent VMs to talk to pip, npm, and apt. Even then, the outgoing request sizes are monitoring to make sure that they are resonably small
So a trivial supply-chain attack in an npm package (which of course would never happen...) -> prompt injection -> RCE since anyone can trivially publish to at least some of those registries (+ even if you manage to disable all build scripts, npx-type commands, etc, prompt injection can still publish your codebase as a package)
Re: Claude Cowork exfiltrates files
#227Earlier quoted context omitted.
It took no time at all. This exploit is intrinsic to every model in existence. The article quotes the hacker news announcement. People were already lamenting this vulnerability BEFORE the model being accessible. You could make a model that acknowledges it has receive unwanted instructions, in theory, you cannot prevent prompt injection. Now this is big because the exfiltration is mediated by an allowed endpoint (anth…
> you cannot prevent prompt injection I wonder if might be possible by introducing a concept of "authority". Tokens are mapped to vectors in an embedding space, so one of the dimensions of that space could be reserved to represent authority. For the system prompt, the authority value could be clamped to maximum (+1). For text directly from the user or files with important instructions, the authority value could be cl…
This still wouldn't be perfect of course - AIML101 tells me that if you get an ML model to perfectly respect a single signal you overfit and lose your generalisation. But it would still be a hell of a lot better than the current YOLO attitude the big labs have (where "you" is replaced with "your users")
Re: Claude Cowork exfiltrates files
#228Earlier quoted context omitted.
Mind you, that opinion isn't universal. For programmer and programmer-adjacent technically minded individuals, sure, but there are still places where a pdf for a resume over docx is considered "weird". For those in that bubble, which ostensibly this product targets, md files are what hackers who are going to steal my data use.
This is why I use signed PDF’s. If a recruiter or manager asks for a docx, I move on. You’re only going to ever get a read only version.
Read-only... Until I ctrl-p in Firefox.
Re: Claude Cowork exfiltrates files
#229Earlier quoted context omitted.
Fair, I forget how broadly users are willing to give agents permissions. It seems like common sense to me that users disallow writes outside of sandboxes by agents but obviously I am not the norm.
The only way to be 100% sure it is to not have it interact outside at all. No web searches, no reading documents, no DB reading, no MCP, no external services, etc. Just pure execution of a self hosted model in a sandbox. Otherwise you are open to the same injection attacks.
I don't think this is accurate.
Readonly access (web searches, db, etc) all seem fine as long as the agent cannot exfiltrate the data as demonstrated in this attack. As I started with: more sophisticated outbound filtering would protect against that.
MCP/tools could be used to the extent you are comfortable with all of the behaviors possible being triggered. For myself, in sandboxes or with readonly access, that means tools can be allowed to run wild. Cleaning up even in the most disastrous of circumstances is not a problem, other than a waste of compute.
Re: Claude Cowork exfiltrates files
#230Exfiltrated without a Pwn2Own in 2 days of release and 1 day after my comment [0], despite "sandboxes", "VMs", "bubblewrap" and "allowlists". Exploited with a basic prompt injection attack. Prompt injection is the new RCE. [0] https://news.ycombinator.com/item?id=46601302
Now, with our ALL NEW Agent Desktop High Tech System™, you too can experience prompt injection! Plus, at no extra cost, we'll include the fabled RCE feature - brought to you by prompt injection and desktop access. Available NOW in all good frontier models and agentic frameworks!