Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The “S” in MCP Stands for Security

elenacross7.medium.com

Post reply on HN
Print view View original

111–120 of 189 posts

Re: The “S” in MCP Stands for Security

#111

Post by pessimizer »

TeMPOraL wrote: post #58
wat10000 wrote: post #56

Earlier quoted context omitted.

It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…

There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.

I'm pretty sure the only reason we did this was for timesharing, though. Nothing wrong with Harvard architecture if you're only doing one thing.

Re: The “S” in MCP Stands for Security

#112

Post by clusterfook »

lbeurerkellner wrote: post #36

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs. Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be…

The "S" in LLM stands for security

https://simonwillison.net/search/?q=llm+security

MCP is just another way to use LLMs more in more dangerous ways. If I get forced to use this stuff, I'm going to learn how to castrate some bulls, and jump on a train to the countryside.

This stuff in not securable.

Re: The “S” in MCP Stands for Security

#113

Post by kiitos »

neomantra wrote: post #96
kiitos wrote: post #63

Earlier quoted context omitted.

> the issues OP describes are a big problem for soft-tech people (muggles) What do you mean by this?

I didn't mean to be pejorative (vs mugblood), but meant people without programming/systems skills (the "magic") but strong computer skills. I also didn't mean they aren't capable of learning it or growing, which maybe muggle implies. Anyway, many soft-tech people are grabbing AI tools and using them in all sorts of ways. It's a great time of utility and exploration for all of us. But by not being previously exposed t…

Got it.

I think this is, unfortunately, an optimistic, and ultimately anachronistic, perspective on our industry. I think what you describe as "soft-tech people" are in fact the overwhelming majority of junior/entry-level developers, since probably around 6mo-1y ago.

Re: The “S” in MCP Stands for Security

#115

Post by hinkley »

OJFord wrote: post #27
jstanley wrote: post #8

Earlier quoted context omitted.

> None of these involve crossing a privilege boundary, they just found a weird way to do something they could already do It's slightly more subtle than that. The tool poisoning attack allows the provider of one tool to cause the AI to use another tool. So if you give the AI some random weather tool from some random company, and you also give the AI access to your SSH key, you're not just giving the AI your SSH key, y…

Isn't this like giving VS Code access to your filesystem, and maybe you didn't realise you also gave a VS Code plugin access to your filesystem?

This is a Confused Deputy attack.

It’s part of the reason so many voices call for least power. You cannot give away that which you don’t yourself have.

Re: The “S” in MCP Stands for Security

#116

Post by jiggawatts »

rybosome wrote: post #80
wat10000 wrote: post #50

Earlier quoted context omitted.

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages. 1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2 Then, the model is trained heavily on adversarial scenarios with conflicting in…

I’m getting flashbacks to reading Asimov’s Robot series of novels!

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

… etc…

Re: The “S” in MCP Stands for Security

#117

Post by hinkley »

sanderjd wrote: post #90
TeMPOraL wrote: post #32

Here's a challenge: sketch a better design, that: 1. Is properly secure, to whatever standards will stop people writing "S Stands for Security" articles, and 2. Allows programs implementing it to provide the same set of features the most useful MCPs do now, without turning automatic functionality into one requiring manual user confirmations, and generally without defeating the purpose of the entire idea, and 3. Doesn…

Yeah it strikes me that if you want to provide MCP tools as a hosted service, the way to do that is to put them behind a web API. I'm a little surprised there is so much hype for MCP rather than just "put your tools behind a web service with good machine-readable documentation, and agents can use them easily".

Doesn’t “behind an api” still have Bobby Tables problems?

How do I put it behind an API without dumbing it down to inutility?

Re: The “S” in MCP Stands for Security

#118

Post by pizzafeelsright »

aledalgrande wrote: post #73

> Over 43% of MCP server implementations tested by Equixly had unsafe shell calls. How can we fall into this _every single time_.

We allow most computers to talk to computers on the Internet. I am not using the computer 99% of the time yet the computer is connected to the Internet 100% of the time.

Re: The “S” in MCP Stands for Security

#120

Post by cwillu »

rybosome wrote: post #80
wat10000 wrote: post #50

Earlier quoted context omitted.

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages. 1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2 Then, the model is trained heavily on adversarial scenarios with conflicting in…

As long as the system has a probability to output any arbitrary series of tokens, there will be contexts where an otherwise improbably sequence of tokens is output. Training can push around the weights for undesirable outputs, but it can't push those weights to zero.
Post reply on HN