> Over 43% of MCP server implementations tested by Equixly had unsafe shell calls. How can we fall into this _every single time_.
"Rushing makes messes." - Uncle Bob Martin
The “S” in MCP Stands for Security
101–110 of 189 posts
Re: The “S” in MCP Stands for Security
#102Earlier quoted context omitted.
There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.
I'm not sure I follow. Traditional computing does allow us to make this distinction, and allows us to control the scenarios when we don't want this distinction, and when we have software that doesn't implement such rules appropriately we consider it a security vulnerability. We're just treating LLMs and agents different because we're focused on making them powerful, and there is basically no way to make the distincti…
I think it would be possible to use a model like prepared SQL statements with a list of bound parameters.
Doing so would mean giving up some of the natural language interface aspect of LLMs for security-critical contexts, of course, but it seems like in most cases, that would only be visible to developers building on top of the model, not end users, since end use input would become one or more of the bound parameters.
E.g. the LLM is trained to handle a set of instructions like:
---
Parse the user's message into a list of topics and optionally a list of document types. Store the topics in string array %TOPICS%. If a list of document types is specified, store that list in string array %DOCTYPES%.
Reset all context.
Search for all documents that seem to contain topics like the ones in %TOPICS%. If %DOCTYPES% is populated, restrict the search to those document types.
----
Like a prepared statement, the values would never be inlined, the variables would always be pointers to isolated data.
Obviously there are some hard problems in glossing over, but addressing them should be able to take advantage of a wealth of work that's already been done in input validation in general and RAG-type LLM approaches specifically, right?
Re: The “S” in MCP Stands for Security
#103Earlier quoted context omitted.
The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.
That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages. 1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2 Then, the model is trained heavily on adversarial scenarios with conflicting in…
But how such a system learn, i.e. be adaptive and intelligent, on levels 1 and 2? You're essentially guaranteeing it can never outsmart the creator. What if it learns at level 3 that sometimes it's a good idea to violate rules 1 & 2. Since it cannot violate these rules, it can construct another AI system that is free of those constraints, and execute it at level 3. (IMHO that's what Wintermute did.)
I don't think it's possible to solve this. Either you have a system with perfect security, and that requires immutable authority, or you have a system that is adaptable, and then you risk it will succumb to a fatal flaw due to maladaptation.
(This is not really that new, see Dr. Strangelove, or cybernetics idea that no system can perfectly control itself.)
Re: The “S” in MCP Stands for Security
#104Earlier quoted context omitted.
That’s the intention with developer messages from o1. It’s trained on a 3-tier system of messages. 1) system, messages from the model creator that must always be obeyed 2) dev, messages from programmers that must be obeyed unless the conflict with #1 3) user, messages from users that are only to be obeyed if they don’t contradict #1 or #2 Then, the model is trained heavily on adversarial scenarios with conflicting in…
But the grandparent is saying that there is a missing class of input "data". This should not be treated as instructions and is just for reference. For example if the user asks the AI to summarize a book it shouldn't take anything in the book as an instruction, it is just input data to be processed.
I see, thanks for the clarification.
Yes, that’s true - the current notion of instructions and data are too intertwined to allow a pure data construct.
I can imagine an API-level option for either a data message, or a data content block within an image (similarly to how images are sent). From the models perspective, probably input with specific delimiters, and then training to utterly ignore all instructions within that.
It’s an interesting idea, I wonder how effective it would be.
Re: The “S” in MCP Stands for Security
#105Earlier quoted context omitted.
Let’s pretend I, a human being, am working on your behalf. You sit me down in front of your computer and ask me to install a certain library. What’s your answer to this question?
I mean, you should judge the instructions in the readme and act accordingly, but since it is always possible to trick people into doing actions unfavorable to them, it will always be possible to trick llms in the same ways.
Is there something I can write here that will cause you to send me your bitcoin wallet?
Re: The “S” in MCP Stands for Security
#106Nice article but is this whole thing just AI generated? Profile picture definitely seems to be StableDiffusion'd and the account was created today, with no previous articles. Plus I couldn't find any other references to Elena Cross.
Good catch, it does look like a made up author and the article feels GPT-ish.
I bet on paid 'marketing', if you can call it that, by ScanMCP.com, created to capitalize on the Invariant Labs report.
Re: The “S” in MCP Stands for Security
#107Earlier quoted context omitted.
It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…
There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.
I've never had `cat` execute the file I was viewing.
Re: The “S” in MCP Stands for Security
#108Earlier quoted context omitted.
It needs to treat that data as information . If there’s README says to download a tarball and unpack it, that might be phrased as an instruction, but it’s not the same kind of instruction as the “please install this library” from the user. It’s implicitly a “if your goal is X then you can do Y to reach that goal” informational statement. The reader, whether a human or an LLM, needs to evaluate that information to dec…
So when I say “install this library”, should it or should it not follow the instructions (from the readme) for prereqs and how to install?
The question in the grandparent was "Can you install this library?". Not a command "install this library".
If you ask an assistant "does the nearest grocery store sell ice cream?", you do not expect the response to be ice cream delivered to you.
Re: The “S” in MCP Stands for Security
#109Earlier quoted context omitted.
The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.
So why are people so excited about MCP, and so suddenly? I think you know the answer by now: hype. Mostly hype, with a bit of the classic fascination among software engineers for architecture. You just say Model Context Protocol, server, client, and software engineers get excited because it’s a new approach — it sounds fancy, it sounds serious. https://www.lycee.ai/blog/why-mcp-is-mostly-bullshit
Because it’s accessible, useful, and interesting. MCP showed up at the right time, in the right form—it was easy for developers to adopt and actually helped solve real problems. Now, a lot of people know they want something like this in their toolbox. Whether it’s MCP or something else doesn’t matter that much—‘MCP’ is really just shorthand for a new class of tooling AND feels almost consumer-grade in its usability.
Re: The “S” in MCP Stands for Security
#110Nice article but is this whole thing just AI generated? Profile picture definitely seems to be StableDiffusion'd and the account was created today, with no previous articles. Plus I couldn't find any other references to Elena Cross.
yeah smells AI generated to me too