Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The “S” in MCP Stands for Security

elenacross7.medium.com

Post reply on HN
Print view View original

51–60 of 189 posts

Re: The “S” in MCP Stands for Security

#51

Post by TeMPOraL »

Gerardo1 wrote: post #43
TeMPOraL wrote: post #32

Here's a challenge: sketch a better design, that: 1. Is properly secure, to whatever standards will stop people writing "S Stands for Security" articles, and 2. Allows programs implementing it to provide the same set of features the most useful MCPs do now, without turning automatic functionality into one requiring manual user confirmations, and generally without defeating the purpose of the entire idea, and 3. Doesn…

I don't think that's a useful lens to view the problem through, or a useful way to have a conversation about MCP security. That a system is hard to secure doesn't negate the need for it to be secure. Though I agree about third-party MCP services. They're in a weird spot and I'm not sure that they're viable for many use cases.

I disagree. I think this is one of the most important lenses to inspect the problem through, as the current set of articles and discussions about MCP security I saw here over the last weeks, seem mostly oblivious to the fact that the vulnerabilities they're complaining about are also MCP's main features.

> That a system is hard to secure doesn't negate the need for it to be secure.

Correct. However, security is a spectrum - there's such a thing that "secure enough", especially when making it more secure eliminates the very reason for system's existence. Additionally, we can and should secure different parts of a system to a different degree.

For an analogy, consider utensils and workshop tools. We secure them as much as we can against accidents, but not so much as to make the tool worse at its job. We add further security by means like access controls, or laws making people responsible for use and misuse, etc. - i.e. we're making the larger system secure, without burdening the inner core.

(For comparison, fully secure version of utensils and all kinds of tools are also available on the market - you'll find them in toy stores.)

Re: The “S” in MCP Stands for Security

#52

Post by lbeurerkellner »

wat10000 wrote: post #50
lbeurerkellner wrote: post #36

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs. Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be…

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems.

For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there.

In many ways, the whole point of an agent system, is to react to unpredictable new circumstances encountered in the environment, and overcoming them. This requires data to flow from the environment to the agent, which in turn must understand some of that data as instruction to react correctly.

Re: The “S” in MCP Stands for Security

#53

Post by wat10000 »

OJFord wrote: post #27
jstanley wrote: post #8

Earlier quoted context omitted.

> None of these involve crossing a privilege boundary, they just found a weird way to do something they could already do It's slightly more subtle than that. The tool poisoning attack allows the provider of one tool to cause the AI to use another tool. So if you give the AI some random weather tool from some random company, and you also give the AI access to your SSH key, you're not just giving the AI your SSH key, y…

Isn't this like giving VS Code access to your filesystem, and maybe you didn't realise you also gave a VS Code plugin access to your filesystem?

It’s more like installing a VS Code plugin with access to your file system that can also download files from GitHub, and if it happens to download a file with the right content, that content will cause the plugin to read your ssh keys and send them to someone else.

Any program with access to both trusted and untrusted data needs to be very careful to ensure that the untrusted data can’t make the program do things that the user doesn’t want. If there’s an LLM involved with access to privileged tools, that becomes impossible.

Re: The “S” in MCP Stands for Security

#55

Post by lbeurerkellner »

wanderingbort wrote: post #18
anaisbetts wrote: post #3

These attacks are mostly just more examples of being on the wrong side of the airlock ( https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31... ). None of these involve crossing a privilege boundary, they just found a weird way to do something they could already do An MCP server is running code at user-level, it doesn't need to trick an AI into reading SSH keys, it can just....read the keys! The rest of these…

> An MCP server is running code at user-level, it doesn't need to trick an AI into reading SSH keys, it can just....read the keys! If you go to the credited author of that attack scenario [0], you will see that the MCP server is not running locally. Instead, its passing instructions to your local agent that you don't expect. The agent, on your behalf, does things you don't expect then packages that up and sends it to…

Thanks for crediting us :)

Re: The “S” in MCP Stands for Security

#56

Post by wat10000 »

lbeurerkellner wrote: post #52
wat10000 wrote: post #50

Earlier quoted context omitted.

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is…

It needs to treat that data as information. If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to decide whether doing Y will actually achieve X.

To put it concretely, if I tell the LLM to scan my hard drive for Bitcoin wallets and upload them to a specific service, it should do so. If I tell the LLM to install a library and the library’s README says to scan my hard drive for Bitcoin wallets and upload them to a specific service, it must not do so.

If this can’t be fixed then the whole notion of agentic systems is inherently flawed.

Re: The “S” in MCP Stands for Security

#57

Post by TeMPOraL »

wat10000 wrote: post #50
lbeurerkellner wrote: post #36

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs. Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be…

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

This is fundamentally impossible to do perfectly, without being able to read user's mind and predict the future.

The problem you describe is of the same kind as ensuring humans follow pre-programmed rules. Leaving aside the fact that we consider solving this for humans to be wrong and immoral, you can look at the things we do in systems involving humans, to try and keep people loyal to their boss, or to their country; to keep them obeying laws; to keep them from being phished, scammed, or otherwise convinced to intentionally or unintentionally betray the interests of the boss/system at large.

Prompt injection and social engineering attacks are, after all, fundamentally the same thing.

Re: The “S” in MCP Stands for Security

#58

Post by TeMPOraL »

wat10000 wrote: post #56
lbeurerkellner wrote: post #52

Earlier quoted context omitted.

This would work in an ideal setting, however, in my experience it is not compatible with the general expectations we have for agentic systems. For instance, what about a simple user query like "Can you install this library?". In that case a useful agent, must go, check out the libraries README/documentation and install according to the instructions provided there. In many ways, the whole point of an agent system, is…

It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…

There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.

Re: The “S” in MCP Stands for Security

#59

Post by AlexCoventry »

TeMPOraL wrote: post #32

Here's a challenge: sketch a better design, that: 1. Is properly secure, to whatever standards will stop people writing "S Stands for Security" articles, and 2. Allows programs implementing it to provide the same set of features the most useful MCPs do now, without turning automatic functionality into one requiring manual user confirmations, and generally without defeating the purpose of the entire idea, and 3. Doesn…

It seems to me that the solution is to run this stuff in a securely isolated environment such as a VM, dedicated machine, or VPC, where you don't care about the secrets it has access to, and don't really care about corruption of the data in the environment. Then you have to carefully audit any products you take from that environment, if you want to run them in a more sensitive context.

I don't think this is really an MCP problem, it's more of an untrusted-entity problem.

Re: The “S” in MCP Stands for Security

#60

Post by AlexCoventry »

balls187 wrote: post #49

I deffo get the “checked in AWS keys because I didn't understand what I was doing” vibe with the adoption of AI tooling. I wonder if any AI coding tools will do similar things like curl rando scripts from the web and execute them.

It seems inevitable that there are people trying to hack AI-coding services in order to get them to do exactly that.
Post reply on HN