21–30 of 91 posts
Seems obvious to me that you should fully vet whatever goes to LLM.
All of these are incredibly obvious. If you have even the slightest idea of what you're doing and review the code before deploying it to prod, this will never succeed. If you have absolutely no idea what you're doing, well, then it doesn't really matter in the end, does it? You're never gonna recognize any security vulnerabilities (as has happened many times with LLM-assisted "no-code" platforms and without any actua…
Well this is wrong. And it's exactly this type of thinking why people will get absolutely burned by this.
First off the fact they chose obvious exploits for explanatory purposes doesn't mean this attack only supports obvious exploits...
And to your second point of "review the code before you deploy to prod", the second attack did not involve deploying any code to prod. It involved an LLM reading a reddit comment or github comment and immediately executing.
People not taking security seriously and waving it off as trivial is what's gonna make this such a terrible problem.
So if you are not careful with your inputs you can get stuff injected. Shouldn't this be very clear from start? With any system you should be careful what you input to it. And consider it as possible vector. Seems obvious to me that you should fully vet whatever goes to LLM.
It is like SQL injection. Probably worse. If you are using unsupervised data for context that ultimately generates executable code you will have this security problem. Duh.
Worse because there's really no equivalent to prepared statements.
What? You run a local LLM for privacy, i.e. because you don't want to share data with $BIGCORP. That has very little to do with the security of the generated code (running in a particular environment).
> The conventional wisdom that local, on-premise models offer a security advantage is flawed. While they provide data privacy, our research shows their weaker reasoning and alignment capabilities make them easier targets for sabotage. Yeah, I'm not following here. If you just run something like deepseek locally, you're going to be okay provided you don't feed it a bogus prompt. Outside of a user copy-pasting a prompt…
>attackers can exploit local LLMs I thought that local LLMs means they run on local computers, without being exposed to the internet. If an attacker can exploit a local LLM, means it already compromised you system and there are better things they can do than trick the LLM to get what they can get directly.
So if you are not careful with your inputs you can get stuff injected. Shouldn't this be very clear from start? With any system you should be careful what you input to it. And consider it as possible vector. Seems obvious to me that you should fully vet whatever goes to LLM.
I get the impression that somehow an attacker is able to inject this prompt (maybe in front of the actual coder’s prompt) in such a way to produce actual production code. I’m waiting to hear how this can happen - cross site attacks on the developer’s browser?
With internal documentation and tickets I think you would have bigger issues... And external documentation. Well maybe there should be tooling to check that. Not expert on MCP. But vetting goes there too.
If you can get malicious instructions into the context of even the most powerful reasoning LLMs in the world you'll still be able to trick them into outputting vulnerable code like this if you try hard enough. I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated att…
If you can get malicious instructions into the context of even the most powerful reasoning LLMs in the world you'll still be able to trick them into outputting vulnerable code like this if you try hard enough. I don't think the fact that small models are easier to trick is particularly interesting from a security perspective, because you need to assume that ANY model can be prompt injected by a suitably motivated att…
Something like "where do we store temporary files the agent creates?" becomes obvious if you have a sandbox you can spin up and down in a couple seconds.