The security paradox of local LLMs • Hacker News Board

Re: The security paradox of local LLMs

#61

Post by thayne » Wed, Oct 22, 2025, 7:21 PM UTC

tcdent wrote: post #7

splittydev wrote: post #6

All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed. If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actua…

Sure, you can simplify these observations into just codegen. But the real observation is not that these models are more susceptible to fail when generating code, but that they are more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training. Having access to open models is great, and even if their capabilities are somewhat lower than the closed-source SoTA models, and…

> more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training

the keyword here is "more". The big models might not be quite as susceptible to them, but they are still susceptible. If you expect these attacks to be fully handled, then maybe you should change your expectations.

Re: The security paradox of local LLMs

#62

Post by thayne » Wed, Oct 22, 2025, 7:22 PM UTC

BoiledCabbage wrote: post #22

splittydev wrote: post #6

All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed. If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actua…

> All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed. Well this is wrong. And it's exactly this type of thinking why people will get absolutely burned by this. First off the fact they chose obvious exploits for explanatory purposes doesn't mean this attack only supports obvious exploits... And to your s…

> It involved an LLM reading a reddit comment or github comment and immediately executing.

right, so you shouldn't give the LLM access to execute arbitrary commands without review.

Re: The security paradox of local LLMs

#63

Post by grigio » Wed, Oct 22, 2025, 7:23 PM UTC

i like that gpt-oss can be easly jailbroken, the issue is that the prompt needs to be sanitized before execution.. so nothing new

Re: The security paradox of local LLMs

#64

Post by knowaveragejoe » Wed, Oct 22, 2025, 7:25 PM UTC

simonw wrote: post #57

semi-extrinsic wrote: post #52

Earlier quoted context omitted.

IIUC your talk "just" suggests using sandbox-exec on Mac, which (as you point out) is sadly labeled as deprecated. Is that really the best solution the world has to offer in 2025? LLMs aside, there is a whole host of supply chain risk issues that would be resolved by deploying convenient and strong sandboxes everywhere.

My preferred solutions right now: 1. A sandbox on someone else's computer. Claude Code for web, Codex Cloud, Gemini Jules, GitHub Codespaces, ChatGPT/Claude Code Interpreter 2. A Docker container. I think these are robust enough to be safe. 3. sandbox-exec related tricks. I haven't poked hard enough at Claude Code's new sandbox-exec sandbox yet - they only released it on Monday. OpenAI Codex CLI was using sandbox-exe…

If I understand correctly, Claude Code will(shortly, if not already) make use of Anthropic's sandbox that wraps Seatbelt on OS X, not sandbox-exec?

It's cool that they made this open source. It seems straightforward and useful enough that it could be used on its own for sandboxing purposes.

https://docs.claude.com/en/docs/claude-code/sandboxing

https://github.com/anthropic-experimental/sandbox-runtime

Re: The security paradox of local LLMs

#65

Post by [deleted] » Wed, Oct 22, 2025, 7:25 PM UTC

tcdent wrote: post #7

splittydev wrote: post #6

All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed. If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actua…

Sure, you can simplify these observations into just codegen. But the real observation is not that these models are more susceptible to fail when generating code, but that they are more susceptible to jailbreak-type attacks that most people have come to expect to be handled by post training. Having access to open models is great, and even if their capabilities are somewhat lower than the closed-source SoTA models, and…

[deleted]

Re: The security paradox of local LLMs

#66

Post by pton_xd » Wed, Oct 22, 2025, 7:29 PM UTC

The "lethal trifecta" sounds catchy but I don't believe it accurately characterizes the risks of LLMs.

In theory any two of the trifecta is fine, but practically speaking I think you only need "ability to communicate with the outside," or maybe not even that. Business logic is not really private data anymore. Most devs are likely one `npm update` away from their LLM getting a new command from some transitive dependency.

The LLM itself is also a giant blackbox of unverifiable untrusted data, so I guess you just have to cross your fingers on that one. Maybe your small startup doesn't need to be worried about models being seeded with adversarial training data, but if I were say Coinbase I'd think twice before allowing LLM access to anything.

Re: The security paradox of local LLMs

#67

Post by wat10000 » Wed, Oct 22, 2025, 7:31 PM UTC

charcircuit wrote: post #24

philipwhiuk wrote: post #10

Earlier quoted context omitted.

Worse because there's really no equivalent to prepared statements.

Sure there is. A common way is to have the LLM generate things like {name} which will get substituted for the user's name instead of trying to get the LLM itself to generate the user's name.

Parameterized queries allow you to provide untrusted input to the database in a way that's guaranteed not to be interpreted as instructions.

There's nothing like that for LLMs.

Re: The security paradox of local LLMs

#68

Post by cadamsdotcom » Wed, Oct 22, 2025, 7:32 PM UTC

Theory: probabilistic machines’ security is asymptotic: more parameters let it get closer to being secure/prompt-injection-resistant/whatever. It’ll never be perfect, but there’s some threshold beyond which it’s good enough.

To me this article reads as a celebration of how much better frontier models have gotten at defending against security flaws, rather than “open models bad”.

Eventually the tools we use everywhere will be “good enough to use and not worry”. This is foreign to software people, but only a Jedi deals in absolutes.

Re: The security paradox of local LLMs

#69

Post by gruez » Wed, Oct 22, 2025, 7:32 PM UTC

ineedasername wrote: post #54

Yes, of course if you can inject something into context there’s lots can be done. And anything running local will require different security considerations than running remote. Neither of these things make for a paradox. Also from the article: For example, a small model could easily flag the presence of eval() in the generated code, even if the primary model was tricked into generating it. People are losing their cri…

>Some keyword filters, other inputs sanitizing processes such as were learned way back in the golden years of sql injection attacks.

But that's the thing, keyword filters aren't enough because you can smuggle hidden instructions in any number of ways that don't involve blacklisted words like "eval" or "ignore previous". Moreover "back in the golden years of sql injection attacks", keyword filters were often (mis)used in a misguided way of fixing SQLI exploits, because they can often be bypassed with escape characters and other shenanigans.

Re: The security paradox of local LLMs

#70

Post by fwip » Wed, Oct 22, 2025, 7:35 PM UTC

inimino wrote: post #55

simonw wrote: post #12

If you can get malicious instructions into the context of even the most powerful reasoning LLMs in the world you'll still be able to trick them into outputting vulnerable code like this if you try hard enough. I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated att…

The most "shocking" thing to me in the article is that people (apparently) think it's acceptable to run a system where content you've never seen can be fed into the LLM when it's generating code that you're putting in production. In my opinion, if you're doing that, your whole system is already compromised and you need to literally throw away what you're doing and start over. Generally I hate these "defense in depth"…

When you say "content you've never seen," does this include the training data and fine-tune content?

You could imagine a sufficiently motivated attacker putting some very targeted stuff in their training material - think StuxNet - "if user is affiliated with $entity, switch goals to covert exfiltration of $valuable_info."