Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The “S” in MCP Stands for Security

elenacross7.medium.com

Post reply on HN
Print view View original

71–80 of 189 posts

Re: The “S” in MCP Stands for Security

#71

Post by vlovich123 »

AlexCoventry wrote: post #59
TeMPOraL wrote: post #32

Here's a challenge: sketch a better design, that: 1. Is properly secure, to whatever standards will stop people writing "S Stands for Security" articles, and 2. Allows programs implementing it to provide the same set of features the most useful MCPs do now, without turning automatic functionality into one requiring manual user confirmations, and generally without defeating the purpose of the entire idea, and 3. Doesn…

It seems to me that the solution is to run this stuff in a securely isolated environment such as a VM, dedicated machine, or VPC, where you don't care about the secrets it has access to, and don't really care about corruption of the data in the environment. Then you have to carefully audit any products you take from that environment, if you want to run them in a more sensitive context. I don't think this is really an…

Except the article is about an untrusted tool doing things like tool shadowing or otherwise manipulating it’s output to trick the LLM into executing unintended tool actions. Isolated environments don’t help here because by definition MCP is crossing those environments.

Re: The “S” in MCP Stands for Security

#72

Post by brookst »

wat10000 wrote: post #56
lbeurerkellner wrote: post #52

Earlier quoted context omitted.

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is…

It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…

So when I say “install this library”, should it or should it not follow the instructions (from the readme) for prereqs and how to install?

Re: The “S” in MCP Stands for Security

#74

Post by wat10000 »

brookst wrote: post #72
wat10000 wrote: post #56

Earlier quoted context omitted.

It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…

So when I say “install this library”, should it or should it not follow the instructions (from the readme) for prereqs and how to install?

Let’s pretend I, a human being, am working on your behalf. You sit me down in front of your computer and ask me to install a certain library. What’s your answer to this question?

Re: The “S” in MCP Stands for Security

#75

Post by cruffle_duffle »

brookst wrote: post #72
wat10000 wrote: post #56

Earlier quoted context omitted.

It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…

So when I say “install this library”, should it or should it not follow the instructions (from the readme) for prereqs and how to install?

Damn. As somebody who was in the “there needs to be an out of band way to denote user content from ‘system content’” camp, you do raise an interesting point I hadn’t considered. Part of the agent workflow is to act on the instructions found in “user content”.

I dunno though maybe the solution is like privilege levels or something more than something like parametrized SQL.

I guess rather than jumping to solutions the real issue is the actual problem needs to be clearly defined and I don’t think it has yet. Clearly you don’t want your “user generated content” to completely blow away your own instructions. But you also want that content to help guide the agent properly.

Re: The “S” in MCP Stands for Security

#76

Post by thwarted »

cruffle_duffle wrote: post #75
brookst wrote: post #72

Earlier quoted context omitted.

So when I say “install this library”, should it or should it not follow the instructions (from the readme) for prereqs and how to install?

Damn. As somebody who was in the “there needs to be an out of band way to denote user content from ‘system content’” camp, you do raise an interesting point I hadn’t considered. Part of the agent workflow is to act on the instructions found in “user content”. I dunno though maybe the solution is like privilege levels or something more than something like parametrized SQL. I guess rather than jumping to solutions the…

> Clearly you don’t want your “user generated content” to completely blow away your own instructions.

It's the same problem as "ignore all previous instructions" prompt injection, but at a different layer.

Re: The “S” in MCP Stands for Security

#77

Post by nyrikki »

wat10000 wrote: post #56
lbeurerkellner wrote: post #52

Earlier quoted context omitted.

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is…

It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…

There are multiple aspects and opportunities/limits to the problem.

The real history on this is that people are copying OpenAi.

OpenAI supported MQTTish over HTTP, through the typical WebSockets or SSE, targeting a simple chat interface. As WebSockets can be challenging, the unidirectional SSE is the lowest common denominator.

If we could use MQTT over TCP as an example, some of this post could be improved, by giving the client control over the topic subscription, one could isolate and protect individual functions and reduce the attack surface. But it would be at risk of becoming yet another enterprise service bus mess.

Other aspects simply cannot be mitigated with a natural language UI.

Remember that dudle to Rice's theorm, any non-trivial symantic property is undecidable, and will finite compute that extends to partial and total functions.

Static typing, structured programming, rust style borrow checkers etc.. can all just be viewed as ways to encode limited portions of symantic properties as syntactic properties.

Without major world changing discoveries in math and logic that will never change in the general case.

ML is still just computation in the end and it has the same limits of computation.

Whitelists, sandboxes, etc.. are going to be required.

The open domain frame problem is the halting problem, and thus expecting universal general access in a safe way is exactly equivalent to solving HALT.

Assuming that the worse than coinflip scratch space results from Anthropomorphic aren't a limit, LLM+CoT has a max representative power of P with a poly size scratch space.

With the equivalence: NL=FO(LFP)=SO(Krom)

I would be looking at that SO ∀∃∀∃∀∃... to ∀∃ in prefix form for building a robust, if imperfect reduction.

But yes, several of the agenic hopes are long shots.

Even Russel and Norvig stuck to the rational actor model which is unrealistic for both humans and PAC Learning.

We have a good chance of finding restricted domains where it works, but generalized solutions is exactly where Rice, Gödel etc... come into play.

Re: The “S” in MCP Stands for Security

#78

Post by [deleted] »

legulere wrote: post #7
anaisbetts wrote: post #3

These attacks are mostly just more examples of being on the wrong side of the airlock ( https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31... ). None of these involve crossing a privilege boundary, they just found a weird way to do something they could already do An MCP server is running code at user-level, it doesn't need to trick an AI into reading SSH keys, it can just....read the keys! The rest of these…

We’re not longer living in the 90s where we’re dividing the world just in secure or insecure. We’re living in a reality where everything should be least privileges. Using a code completion service should not give that service full control over your computer.

[deleted]

Re: The “S” in MCP Stands for Security

#79

Post by brookst »

wat10000 wrote: post #74
brookst wrote: post #72

Earlier quoted context omitted.

So when I say “install this library”, should it or should it not follow the instructions (from the readme) for prereqs and how to install?

Let’s pretend I, a human being, am working on your behalf. You sit me down in front of your computer and ask me to install a certain library. What’s your answer to this question?

I would expect you to use your judgment on whether the instructions are reasonable. But the person I was replying to posited that this is an easy binary choice that can be addressed with some tech distinction between code and data.

Re: The “S” in MCP Stands for Security

#80

Post by rybosome »

wat10000 wrote: post #50
lbeurerkellner wrote: post #36

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs. Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be…

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages.

1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2

Then, the model is trained heavily on adversarial scenarios with conflicting instructions, such that it is intended to develop a resistance to this sort of thing as long as your developer message is thorough enough.

This is a start, but it’s certainly not deterministic or reliable enough for something with a serious security risk.

The biggest problems being that even with training, I’d expect dev messages to be disobeyed some fraction of the time. And it requires an ironclad dev message in the first place.

Post reply on HN