Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#5

Post by hu3 » Fri, May 23, 2025, 11:34 AM UTC

cedws wrote: post #4

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

I would have the same caution, if my code was any special.

But the reality is I'm very well compensated to summon CRUD slop out of thin air. It's well tested though.

I wish good luck to those who steal my code.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#6

Post by mdaniel » Fri, May 23, 2025, 3:12 PM UTC

Running Duo as a system user was crazypants and I'm sad that GitLab fell into that trap. They already have personal access tokens so even if they had to silently create one just for use with Duo that would be a marked improvement over giving an LLM read access to every repo in the platform

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#7

Post by mdaniel » Fri, May 23, 2025, 3:15 PM UTC

hu3 wrote: post #5

cedws wrote: post #4

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

I would have the same caution, if my code was any special. But the reality is I'm very well compensated to summon CRUD slop out of thin air. It's well tested though. I wish good luck to those who steal my code.

You say code as if the intellectual property is the thing an attacker is after, but my experience has been that folks often put all kinds of secrets in code thinking that the "private repo" is a strong enough security boundary

I absolutely am not implying you are one of them, merely that the risk is not the same for all slop crud apps universally

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#8

Post by wunderwuzzi23 » Fri, May 23, 2025, 5:56 PM UTC

Great work!

Data leakage via untrusted third party servers (especially via image rendering) is one of the most common AI Appsec issues and it's concerning that big vendors do not catch these before shipping.

I built the ASCII Smuggler mentioned in the post and documented the image exfiltration vector on my blog as well in past with 10+ findings across vendors.

GitHub Copilot Chat had a very similar bug last year.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#9

Post by aestetix » Fri, May 23, 2025, 10:52 PM UTC

Does that mean Gitlab Duo can run Doom?

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#10

Post by danpalmer » Mon, May 26, 2025, 4:50 AM UTC

cedws wrote: post #4

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

Prompt injection is unlikely to be fixed. I'd stop thinking about LLMs as software where you can with enough effort just fix a SQL injection vulnerability, and start thinking about them like you'd think about insider risk from employees.

That's not to say that they are employees or perform at that level, they don't, but it's to say that LLM behaviours are fuzzy and ill-defined, like humans. You can't guarantee that your users won't click on a phishing email – you can train them, you can minimise risk, but ultimately you have to have a range of solutions applied together and some amount of trust. If we think about LLMs this way I think the conversation around security will be much more productive.