161–170 of 227 posts
Earlier quoted context omitted.
This isn't a bug in the LLMs. It's a bug in the software that uses those LLMs. An LLM on its own can't execute code. An LLM harness like Antigravity adds that ability, and if it does it carelessly that becomes a security vulnerability.
No matter how many prompt changes you make it won't be possible to fix this.
Earlier quoted context omitted.
The problem is a bit wider than that. One can frame it as "google gemini is vulterable" or "google's new VS code clone is vulnerable". The bigger picture is that the model predicts tokens (words) based on all the text it have. In a big codebase it becomes exponentially easier to mess the model's mind. At some point it will become confused what is his job. What is part of the "system prompt" and "code comments in the…
I'm not sure what your argument is here. We shouldn't be making a fuss about all these prompt injection attacks because they're just inevitable so don't worry about it? Or we should stop being surprised that this happens because it happens all the time? Either way I would be extremely concerned about these use cases in any circumstance where the program is vulnerable and rapid, automatic or semi-automatic updates are…
The argument is: we need to be careful about how LLMs are integrated with tools and about what capabilities are extended to "agents". Much more careful than what we currently see.
I'm not quite convinced. You're telling the agent "implement what it says on " and the blog is malicious and exfiltrates data. So Gemini is simply following your instructions. It is more or less the same as running "npm install " on your own. Ultimately, AI or not, you are the one responsible for validating dependencies and putting appropriate safeguards in place.
1. There are countless ways to hide machine-readable content on the blog that doesn't make a visible impact on the page as normally viewed by humans.
2. Even if you somehow verify what the LLM will see, you can't trivially predict how it will respond to what it sees there.
3. In particular, the LLM does not make a proper distinction between things that you told it to do, and things that it reads on the blog.
That's the bleeding edge you get with vibe coding
cutting edge perhaps?
I feel like I'm going insane reading how people talk about "vulnerabilities" like this. If you give an llm access to sensitive data, user input and the ability to make arbitrary http calls it should be blindingly obvious that it's insecure. I wouldn't even call this a vulnerability, this is just intentionally exposing things. If I had to pinpoint the "real" vulnerability here, it would be this bit, but the way it's j…
These aren't vulnerabilities in LLMs. They are vulnerabilities in software that we build on top of LLMs. It's important we understand them so we can either build software that doesn't expose this kind of vulnerability or, if we build it anyway, we can make the users of that software aware of the risks so they can act accordingly.
People don't think of this as a risk when they're building the software, either because they just don't think about security at all, or because they mentally model the LLM as unerringly subservient to the user — as if we'd magically solved the entire class of philosophical problems Asimov pointed out decades ago without even trying.
Sooner or later I believe, there will be models which can be deployed locally on your mac and are as good as say Sonnet 4.5. People should shift to completely local at that point. And use sandbox for executing code generated by llm. Edit: "completely local" meant not doing any network calls unless specifically approved. When llm calls are completely local you just need to monitor a few explicit network calls to be su…
The problem is that people want the agent to be able to do "research" on the fly.
Earlier quoted context omitted.
Sorry, I didn't elaborate. But "completely local" meant not doing any network calls unless specifically approved. When llm calls are completely local you just need to monitor a few explicit network calls to be sure.
In a realistic and useful scenario, how would you approve or deny network calls made by a LLM?
You would have to design the other system to require approval when it sees a request. But this of course still relies on the human to understand those requests. And will presumably become tedious and susceptible to consent fatigue.
Absolute amateurs.
I really liked Simon's Willison's [1] and Meta's [2] approach using the "Rule of Two". You can have no more than 2 of the following: - A) Process untrustworthy input - B) Have access to private data - C) Be able to change external state or communicate externally. It's not bullet-proof, but it has helped communicate to my management that these tools have inherent risk when they hit all three categories above (and any…