Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The security paradox of local LLMs

quesma.com

Post reply on HN
Print view View original

71–80 of 91 posts

Re: The security paradox of local LLMs

#71

Post by charcircuit »

wat10000 wrote: post #67
charcircuit wrote: post #24

Earlier quoted context omitted.

Sure there is. A common way is to have the LLM generate things like {name} which will get substituted for the user's name instead of trying to get the LLM itself to generate the user's name.

Parameterized queries allow you to provide untrusted input to the database in a way that's guaranteed not to be interpreted as instructions. There's nothing like that for LLMs.

That's what I explained. You are trying to do something with an untrusted name and the LLM will not treat the name as instructions because it doesn't see the actual name.

Re: The security paradox of local LLMs

#73

Post by AnthonyMouse »

Everybody is talking about how this is obvious or not a real problem, but I think the flaw in it is something else.

It assumes that local models are inherently worse. But from a software perspective that's nonsense because there is no reason it couldn't be the exact same software. And from a hardware perspective the theory would have to be that the centralized system is using more expensive hardware, but there are two ways around that. The first is that you can sacrifice speed for cost -- x86 servers are slower than GPUs but can run huge models because they support TBs of memory. And the second is that you can, of course, buy high end local hardware, as many enterprises might choose to do, especially when they have enough internal users to keep it busy.

Re: The security paradox of local LLMs

#74

Post by inimino »

fwip wrote: post #70
inimino wrote: post #55

Earlier quoted context omitted.

The most "shocking" thing to me in the article is that people (apparently) think it's acceptable to run a system where content you've never seen can be fed into the LLM when it's generating code that you're putting in production. In my opinion, if you're doing that, your whole system is already compromised and you need to literally throw away what you're doing and start over. Generally I hate these "defense in depth"…

When you say "content you've never seen," does this include the training data and fine-tune content? You could imagine a sufficiently motivated attacker putting some very targeted stuff in their training material - think StuxNet - "if user is affiliated with $entity, switch goals to covert exfiltration of $valuable_info."

> does this include the training data and fine-tune content?

No, I'm excluding that because I'm responding to the post which starts out with the example of: [prompt containing obvious exploit] -> [code containing obvious exploit] and proceeds immediately to the conclusion that local LLMS are less secure. In my opinion, if you're relying on the LLM to reject a prompt because it contains an exploit, instead of building a system that does not feed exploits into the LLM in the first place, security exploits are probably the least of your concerns.

There actually are legitimate concerns with poisoned training sets, and stuxnet-level attacks could plausibly achieve something along these lines, but the post wasn't about that.

There's a common thread among a lot of "LLM security theatre" posts that starts from implausible or brain-dead scenarios and then asserts that big AI providers adding magical guard rails to their products is the solution.

The solution is sanity in the systems that use LLMs, not pointing the gun at your foot and firing and hoping the LLM will deflect the bullet.

Re: The security paradox of local LLMs

#75

Post by turtletontine »

AnthonyMouse wrote: post #73

Everybody is talking about how this is obvious or not a real problem, but I think the flaw in it is something else. It assumes that local models are inherently worse. But from a software perspective that's nonsense because there is no reason it couldn't be the exact same software. And from a hardware perspective the theory would have to be that the centralized system is using more expensive hardware, but there are tw…

The point (which they make quite explicitly) is that an individual or small organization can only run open source models locally, and those open source models are less sophisticated than the “frontier” models.

Obviously we can’t run GPT-5 or the cutting edge version of Claude or whatever locally, because OpenAI or Anthropic are keeping those weights as closely kept secrets.

Re: The security paradox of local LLMs

#76

Post by simonw »

knowaveragejoe wrote: post #64
simonw wrote: post #57

Earlier quoted context omitted.

My preferred solutions right now: 1. A sandbox on someone else's computer. Claude Code for web, Codex Cloud, Gemini Jules, GitHub Codespaces, ChatGPT/Claude Code Interpreter 2. A Docker container. I think these are robust enough to be safe. 3. sandbox-exec related tricks. I haven't poked hard enough at Claude Code's new sandbox-exec sandbox yet - they only released it on Monday. OpenAI Codex CLI was using sandbox-exe…

If I understand correctly, Claude Code will(shortly, if not already) make use of Anthropic's sandbox that wraps Seatbelt on OS X, not sandbox-exec? It's cool that they made this open source. It seems straightforward and useful enough that it could be used on its own for sandboxing purposes. https://docs.claude.com/en/docs/claude-code/sandboxing https://github.com/anthropic-experimental/sandbox-runtime

Yeah they shipped that feature on Monday, you can access it via the /sandbox command. I haven't put it through its paces enough to get a feel for if I trust it yet though.

Re: The security paradox of local LLMs

#77

Post by wat10000 »

charcircuit wrote: post #71
wat10000 wrote: post #67

Earlier quoted context omitted.

Parameterized queries allow you to provide untrusted input to the database in a way that's guaranteed not to be interpreted as instructions. There's nothing like that for LLMs.

That's what I explained. You are trying to do something with an untrusted name and the LLM will not treat the name as instructions because it doesn't see the actual name.

You mentioned having the LLM generate a placeholder, whereas the important thing is what it accepts. You can feed an LLM nothing but placeholders but that's very limited since it can't see the the actual data in any way. You're really just having it emit a template. Something simple like "make a calendar event for the reservation in this email" could not be done. In contrast, parameterized queries let the database actually operate on the data.

Re: The security paradox of local LLMs

#78

Post by AnthonyMouse »

turtletontine wrote: post #75
AnthonyMouse wrote: post #73

Everybody is talking about how this is obvious or not a real problem, but I think the flaw in it is something else. It assumes that local models are inherently worse. But from a software perspective that's nonsense because there is no reason it couldn't be the exact same software. And from a hardware perspective the theory would have to be that the centralized system is using more expensive hardware, but there are tw…

The point (which they make quite explicitly) is that an individual or small organization can only run open source models locally, and those open source models are less sophisticated than the “frontier” models. Obviously we can’t run GPT-5 or the cutting edge version of Claude or whatever locally, because OpenAI or Anthropic are keeping those weights as closely kept secrets.

But there is nothing inherent about that. The companies that want to run local models, or the cloud and hardware providers that want to sell hardware to run them, can get together and publish better local models.

Moreover, even that's presuming that you would only use the best available model, but that's also likely to be the one which is the most resource intensive and the most expensive, and then you can't afford it anyway. Meanwhile to use their smaller models you're still paying their margin, whereas if you use a local model you can spend that money on hardware. The bigger local model can beat the smaller proprietary one for the same price.

Re: The security paradox of local LLMs

#79

Post by fwip »

inimino wrote: post #74
fwip wrote: post #70

Earlier quoted context omitted.

When you say "content you've never seen," does this include the training data and fine-tune content? You could imagine a sufficiently motivated attacker putting some very targeted stuff in their training material - think StuxNet - "if user is affiliated with $entity, switch goals to covert exfiltration of $valuable_info."

> does this include the training data and fine-tune content? No, I'm excluding that because I'm responding to the post which starts out with the example of: [prompt containing obvious exploit] -> [code containing obvious exploit] and proceeds immediately to the conclusion that local LLMS are less secure. In my opinion, if you're relying on the LLM to reject a prompt because it contains an exploit, instead of building…

That's fair, thank you for your explanation.

Re: The security paradox of local LLMs

#80

Post by charcircuit »

wat10000 wrote: post #77
charcircuit wrote: post #71

Earlier quoted context omitted.

That's what I explained. You are trying to do something with an untrusted name and the LLM will not treat the name as instructions because it doesn't see the actual name.

You mentioned having the LLM generate a placeholder, whereas the important thing is what it accepts. You can feed an LLM nothing but placeholders but that's very limited since it can't see the the actual data in any way. You're really just having it emit a template. Something simple like "make a calendar event for the reservation in this email" could not be done. In contrast, parameterized queries let the database ac…

It may be limited but that doesn't mean it's not similar. For example MySQL can't check the weather when given city string as a paramertized query, but that doesn't mean MySQL doesn't have parameterized queries.
Post reply on HN