31–40 of 61 posts
Earlier quoted context omitted.
AI labs currently have no solution for this problem and have you shoulder the risk for it.
Evidence?
In the meantime security researchers are publishing proof of concept data exfiltration attacks all the time. I've been collecting those here: https://simonwillison.net/tags/exfiltration-attacks/
Earlier quoted context omitted.
The risk isn't from the AI labs. It's from malicious attackers who sneak instructions to coding agents that cause them to steal your data, including your environment variable secrets - or cause them to perform destructive or otherwise harmful actions using the permissions that you've granted to them.
Simon, I know you're the AI bigwig but I'm not sure that's correct. I know that's the "story" (but maybe just where the AI labs would prefer we look?). How realistic is it really that MCP/tools/web search is being corrupted by people to steal prompts/convos like this? I really think this is such low prop. And if it does happen, the flaw is the AI labs for letting something like this occur. Respect for your writing, b…
That doesn't mean the risk isn't there - it means malicious actors have not yet started exploiting it.
Johann Rehberger calls this effect "The Normalization of Deviance in AI", borrowing terminology from the 1986 Space Shuttle Challenger disaster report: https://embracethered.com/blog/posts/2025/the-normalization-...
Short version: the longer a company or community gets away with behaving in an unsafe way without feeling the consequences, the more they are likely to ignore those risks.
I'm certain that's what is happening to us all today with coding agents. I use them in an unsafe way myself.
I think people's focus on the threat model from AI corps is wrong. They are not going to "steal your precious SSH/cloud/git credentials" so they can secretly poke through your secret-sauce, botnet your servers or piggy back off your infrastructure, lol of lols. Similarly the possibility of this happening from MCP tool integrations is overblown. This dangerous misinterpretation of the actual possible threats simply be…
Isn’t this (part of) the point of MCP.
Possibly, but the point is that MCP is a DOA idea. An agent, like Claude code or opencode, don’t need an MCP. it’s nonsensical to expect or need an MCP before someone can call you. There is no `git` MCP either . Opencode is fully capable of running `git add .` or `aws ec2 terminate-instance …` or `curl -XPOST https://…` Why do we need the MCP? The problem now is that someone can do a prompt injection to tell it to se…
> Why do we need the MCP?
> The problem now
And there it is.
I understand that this is an alternative solution, and appreciate it.
[flagged]
How would that work? If the AI can use it, it can read it. E.g:
secret-store "foo" > file
cat file
https://www.whatismybrowser.com/detect/what-http-headers-is-...
Basically, for every "use" of a secret, you'd have to write a dedicated application which performs that task in a secure manner. It's not just the case of adding a special secret store.
[flagged]
Is this a reimplementation of Fly.io’s Tokenizer? How does it compare? https://fly.io/blog/tokenized-tokens/ https://github.com/superfly/tokenizer
I'm working on something similar called agent-creds [0]. I'm using Envoy as the transparent (MITM) proxy and macaroons for credentials. The idea is that you can arbitrarily scope down credentials with macaroons, both in terms of scope (only certain endpoints) and time. This really limits the damage that an agent can do, but also means that if your credentials are leaked they are already expired within a few minutes.…
[flagged]
We spent the last 50 years of computer security getting to a point where we keep sensitive credentials out of the hands of humans. I guess now we have to take the next 50 years to learn the lesson that we should keep those same credentials out of the hands of LLMs as well?
I'll be sitting on the sideline eating popcorn in that case.