Live data from Hacker News

The “S” in MCP Stands for Security

elenacross7.medium.com

131–140 of 189 posts

Re: The “S” in MCP Stands for Security

#131

Earlier quoted context omitted.

You never accidentally cat-ed a binary file and borked your terminal? If not, then find some random binary - an image, archive, maybe even /dev/random - and cat it. Hint: `reset` will fix the terminal afterwards. Usually.

That's not the same thing, and hasn't been a security issue for quite a while now.

It is the same thing, that's the point. It all depends on how you look at it.

Most software is trying to enforce a distinction between "code" and "data", in the sense that whatever we call "data" can only cause very limited set of things to happen - but that's just the program rules that make this distinction, fundamentally it doesn't exist. And thus, all it takes is some little bug in your input parser, or in whatever code interprets[0] that data, and suddenly data becomes code.

See also: most security vulnerabilities that ever existed.

Or maybe an example from the opposite end will be illuminating. Consider WMF/EMF family of image formats[1], that are notable for handling both raster and vector data well. The interesting thing about WMF/EMF files is that the data format itself is... serialized list of function calls to Window's GDI+ API.

(Edit: also, hint: look at the abstraction layers. Your, say, Python program is Python code, but for the interpreter, it's merely data; your Python interpreter itself is merely data for the layer underneath, and so on, and so on.)

You can find countless examples of the same information being code or data in all kinds of software systems - and outside of them, too; anything from music players to DNA. And, going all the way up to theoretical: there is no such thing in nature as "code" distinct from "data". There is none, there is no way to make that distinction, atoms do not carry such property, etc. That distinction is only something we do for convenience, because most of the time it's obvious for us what is code and what is data - but again, that's not something in objective reality, it's merely a subjective opinion.

Skipping the discussion about how we make code/data distinction work (hint: did you prove your data as processed by your program isn't itself a Turing-complete language?) - the "problem" with LLMs is that we expect them to behave with human-like, fully general intelligence, processing all inputs together as a single fused sensory stream. There is no way to introduce a provably perfect distinction between "code" and "data" here without losing some generality in the model.

And you definitely ain't gonna do it with prompts - if one part of the input can instruct the model to do X, another can always make it disregard X. It's true for humans too. Helpful example: imagine you're working a data-entry job; you're told to retype a binder of text into your terminal as-is, ignoring anything the text actually says (it's obviously data). Halfway through the binder, you hit on a part of text that reads as a desperate plea for help from kidnapped slave worker claiming to have produced the data you're retyping, and who's now begging you to tell someone, call police, etc. Are you going to ignore it, just because your boss said you should ignore contents of the data you're transcribing? Are you? Same is going to be true for LLMs - sufficiently convincing input will override whatever input came before.

--

[0] - Interpret, interpreter... - that should in itself be a hint.

[1] - https://en.wikipedia.org/wiki/Windows_Metafile

Re: The “S” in MCP Stands for Security

#132

Earlier quoted context omitted.

There is no hard distinction between "code" and "data". Both are the same thing. We've built an entire computing industry on top of that fact, and it sort of works, and that's all with most software folks not even being aware that whether something is code or data is just a matter of opinion.

And yet the distinction must be made. Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” Untrusted data must never be executed as code in a privileged context. When there’s a way to make that happen, it’s considered a serious flaw that must be fixed.

> Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.”

What about being treated as code when it's supposed to be?

(What is the difference between code execution vulnerability and a REPL? It's who is using it.)

Whatever you call program vs. its data, the program can always be viewed as an interpreter for a language, and your input as code in that language.

See also the subfield of "langsec", which is based on this premise, as well as the fact that you probably didn't think of that and thus your interpreter/parser is implicitly spread across half your program (they call it "shotgun parser"), and your "data" could easily be unintentionally Turing-complete without you knowing :).

EDIT:

I swear "security" is becoming a cult in our industry. Whether or not you call something "security vulnerability" and therefore "a problem", doesn't change the fundamental nature of this thing. And the fundamental nature of information is, there exist no objective, natural distinction between code and data. It can be drawn arbitrarily, and systems can be structured to emulate it - but that still just means it's a matter of opinion.

EDIT2: Not to mention, security itself is not objective. There is always the underlying assumption - the answer to a question, who are you protecting the system from, and for who are you doing it?. You don't need to look far to find systems where users are seen in part as threat actors, and thus get disempowered in the name of protecting the interests of vendor and some third parties (e.g. advertisers).

Re: The “S” in MCP Stands for Security

#133

Earlier quoted context omitted.

I mean, you should judge the instructions in the readme and act accordingly, but since it is always possible to trick people into doing actions unfavorable to them, it will always be possible to trick llms in the same ways.

Is there something I can write here that will cause you to send me your bitcoin wallet?

There probably is, but you're also probably not smart enough (and probably no one is) to figure out what it is.

But it does happens, in very similar circumstances (twitter, e-mail) very regularly.

Re: The “S” in MCP Stands for Security

#134

Earlier quoted context omitted.

I mean, you should judge the instructions in the readme and act accordingly, but since it is always possible to trick people into doing actions unfavorable to them, it will always be possible to trick llms in the same ways.

Is there something I can write here that will cause you to send me your bitcoin wallet?

Many technically adept people on HN acknowledge that they would be vulnerable to a carefully targeted spear phishing attack.

The idea that it would be carried out beginning in a post on HN is interesting, but to me kind of misses the main point... which is the understanding that everyone is human, and the right attack at the right time (plus a little bad luck) could make them a victim.

Once you make it a game, stipulating that your spear phishing attack is going to begin with an interesting response on HN, it's fun to let your imagination unwind for a while.

Re: The “S” in MCP Stands for Security

#136

Earlier quoted context omitted.

I'm not sure I follow. Traditional computing does allow us to make this distinction, and allows us to control the scenarios when we don't want this distinction, and when we have software that doesn't implement such rules appropriately we consider it a security vulnerability. We're just treating LLMs and agents different because we're focused on making them powerful, and there is basically no way to make the distincti…

I think it would be possible to use a model like prepared SQL statements with a list of bound parameters. Doing so would mean giving up some of the natural language interface aspect of LLMs for security-critical contexts, of course, but it seems like in most cases, that would only be visible to developers building on top of the model, not end users, since end use input would become one or more of the bound parameters…

The LLM ultimately needs to see the actual text in %TOPICS% etc, meaning that it must be somewhere in its input.

Re: The “S” in MCP Stands for Security

#137

The post highlights and cites a few attack scenarios we originally described in a security note (tool poisoning, shadowing, MCP rug pull), published a few days ago [1]. I am the author of said blog post at Invariant Labs. Different from what many suspect, the security problem with MCP-style LLM tool calling is not in isolating different MCP server implementations. MCP server implementations that run locally should be…

The fact that all LLM input gets treated equally seems like a critical flaw that must be fixed before LLMs can be given control over anything privileged. The LLM needs an ironclad distinction between “this is input from the user telling me what to do” and “this is input from the outside that must not be obeyed.” Until that’s figured out, any attempt at security is going to be full of holes.

There is no way to fix it. It's part of the basic architecture of LLMs.

Re: The “S” in MCP Stands for Security

#138

Earlier quoted context omitted.

Is there something I can write here that will cause you to send me your bitcoin wallet?

Many technically adept people on HN acknowledge that they would be vulnerable to a carefully targeted spear phishing attack. The idea that it would be carried out beginning in a post on HN is interesting, but to me kind of misses the main point... which is the understanding that everyone is human, and the right attack at the right time (plus a little bad luck) could make them a victim. Once you make it a game, stipul…

The thing is, an LLM agent could be subverted with an HN comment pretty easily, if its task happened to take it to HN.

Yes, humans have this general problem too, but they’re far less vulnerable to it.

Re: The “S” in MCP Stands for Security

#139

Earlier quoted context omitted.

And yet the distinction must be made. Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” Untrusted data must never be executed as code in a privileged context. When there’s a way to make that happen, it’s considered a serious flaw that must be fixed.

> Do you know what it’s called when data is treated as code when it’s not supposed to be? It’s called a “security vulnerability.” What about being treated as code when it's supposed to be? (What is the difference between code execution vulnerability and a REPL? It's who is using it .) Whatever you call program vs. its data, the program can always be viewed as an interpreter for a language, and your input as code in t…

Imagine your browser had a flaw I could exploit by carefully crafting the contents this comment, which allows me to take over your computer. You’d consider that a serious problem, right? You’d demand a quick fix from the browser maker.

Now imagine that there is no fix because the ability for a comment to take control of the whole thing is an inherent part of how it works. That’s how LLM agents are.

If you have an LLM agent that can read your email and read the web then you have an agent which can pretty easily be made to leak the contents of your private emails to me.

Yes, your email program may actually have a vulnerability which allows this to happen, with no LLM involved. The difference is, if there is such a vulnerability then it can be fixed. It’s a bug, not an inherent part of how the program works.

Re: The “S” in MCP Stands for Security

#140

Earlier quoted context omitted.

Many technically adept people on HN acknowledge that they would be vulnerable to a carefully targeted spear phishing attack. The idea that it would be carried out beginning in a post on HN is interesting, but to me kind of misses the main point... which is the understanding that everyone is human, and the right attack at the right time (plus a little bad luck) could make them a victim. Once you make it a game, stipul…

The thing is, an LLM agent could be subverted with an HN comment pretty easily, if its task happened to take it to HN. Yes, humans have this general problem too, but they’re far less vulnerable to it.

Yes, I agree. My point was more about the current way we do LLM agents where they are essentially black box that act on text.

By design it can output anything given the right input.

This approach will always be vulnerable in the ways we talk about here, we can only up the guardrails around it.

I think one of the best ways to have truly secure AI agents is to do better natural language AIs that are far less blackbox-y.

But I don't know enough about progress on this side.

Post reply on HN