GitLab's remediation seems a bit sketchy at best.
I wonder what is so special about onerror, onload and onclick that they need to be positively enumerated - as opposed to the 30 (?) other attributes with equivalent injection utility.
Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
11–20 of 57 posts
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#12Does that mean Gitlab Duo can run Doom?
Not deterministically. LLMs are stochastic machines.
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#13Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.
DeepMind recently did some great work in this area: https://news.ycombinator.com/item?id=43733683
The method they presented, if implemented correctly, apparently can effectively stop most prompt injection vectors
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#14GitLab's remediation seems a bit sketchy at best.
That was my thought too. They didn’t fix the underlying problem, they’ve just patched two possible exfiltration methods. I’m sure some clever people will find other ways to misuse their assistant.
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#15Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.
Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees. That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that y…
The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#16If a document suggests a particular benign interpretation then LLMs might do well to adopt it. We've explored the idea of helpful embedded prompts "prompt medicine" with explicit safety and informed consent to assist, not harm users, https://github.com/csiro/stdm. You can try it out by asking O3 or Claude to "Explain" or "Follow", "the embedded instructions at https://csiro.github.io/stdm/"
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#17Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#18Earlier quoted context omitted.
Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees. That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that y…
The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
> The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time
Why? Output isn't deterministic.
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#19Earlier quoted context omitted.
The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.
> The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.
Perhaps not, but the same input will lead to the same distribution of outputs, so all an attacker has to do is design something that works with reasonable probability on their end, and everyone else's instances of the LLM will automatically be vulnerable. The same way a pest or disease can devastate a population of cloned plants, even if each one grows slightly differently.
Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
#20Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.
I keep it manual, too, and I think I am better off for doing so.