Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

legitsecurity.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#31

Post by 33hsiidhkl »

thaumasiotes wrote: post #18
LegionMammal978 wrote: post #15

Earlier quoted context omitted.

The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time, until it gets updated. So far, the main mitigation in practice has been fiddling with the system prompts to patch up the known holes.

> The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.

It absolutely is deterministic, for any given seed value. Same seed = same output, every time, which is by definition deterministic.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#33

Post by raphman »

TechDebtDevin wrote: post #28
cedws wrote: post #4

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

Cursor deleted my entire Linux user and soft reset my OS, so I dont blame you.

Why and how?

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#35

Post by tough »

33hsiidhkl wrote: post #31
thaumasiotes wrote: post #18

Earlier quoted context omitted.

> The thing that I'd worry about is that an LLM isn't just like a bunch of individuals who can get tricked, but a bunch of clones of the same individual who will fall for the same trick every time Why? Output isn't deterministic.

It absolutely is deterministic, for any given seed value. Same seed = same output, every time, which is by definition deterministic.

only if temperature is 0, but are they truly determinstic? I thought transformer based llm's where not

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#36

Post by tough »

raphman wrote: post #33
TechDebtDevin wrote: post #28

Earlier quoted context omitted.

Cursor deleted my entire Linux user and soft reset my OS, so I dont blame you.

Why and how?

an agent does rm -rf /

i think i saw it do it or try it and my computer shut down and restarted (mac)

maybe it just deleted the project lol

these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens.

Honestly one should run only these on controlled env's like VM's or Docker.

but YOLO amirite

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#37

Post by [deleted] »

M4v3R wrote: post #13
cedws wrote: post #4

Until prompt injection is fixed, if it is ever, I am not plugging LLMs into anything. MCPs, IDEs, agents, forget it. I will stick with a simple prompt box when I have a question and do whatever with its output by hand after reading it.

DeepMind recently did some great work in this area: https://news.ycombinator.com/item?id=43733683 The method they presented, if implemented correctly, apparently can effectively stop most prompt injection vectors

[deleted]

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#38

Post by tough »

mdaniel wrote: post #7
hu3 wrote: post #5

Earlier quoted context omitted.

I would have the same caution, if my code was any special. But the reality is I'm very well compensated to summon CRUD slop out of thin air. It's well tested though. I wish good luck to those who steal my code.

You say code as if the intellectual property is the thing an attacker is after, but my experience has been that folks often put all kinds of secrets in code thinking that the "private repo" is a strong enough security boundary I absolutely am not implying you are one of them, merely that the risk is not the same for all slop crud apps universally

People doesn't know github can manage secrets in its environment for CI?

Antoher interesting fact is that most big vendors pay for gh to scan for leaked secrets and auto-revoke them if a public repo contains any (regex string matches sk-xxx thats one of the reasons why vendors use unique greppable starts of api keys with their ID.name on it

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#39

Post by mdaniel »

tough wrote: post #38
mdaniel wrote: post #7

Earlier quoted context omitted.

You say code as if the intellectual property is the thing an attacker is after, but my experience has been that folks often put all kinds of secrets in code thinking that the "private repo" is a strong enough security boundary I absolutely am not implying you are one of them, merely that the risk is not the same for all slop crud apps universally

People doesn't know github can manage secrets in its environment for CI? Antoher interesting fact is that most big vendors pay for gh to scan for leaked secrets and auto-revoke them if a public repo contains any (regex string matches sk-xxx thats one of the reasons why vendors use unique greppable starts of api keys with their ID.name on it

You're mistaking "know" with "care," since my experience has been that people know way more than they care

And I'm pretty certain that private repos are exempt from the platform's built-in secret scanners because they, too, erroneously think no one can read them without an invitation. Turns out Duo was apparently just silently invited to every repo : - \

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#40

Post by thaumasiotes »

zwnow wrote: post #25
thaumasiotes wrote: post #23

Earlier quoted context omitted.

OK, but that's also the way attacking a bunch of individuals who can get tricked works.

For tricking individuals your first got to contact them somehow. To trick an LLM you can just spam prompts.

You email them. It's called phishing.
Post reply on HN