Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

legitsecurity.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#41

Post by 33hsiidhkl »

tough wrote: post #35
33hsiidhkl wrote: post #31

Earlier quoted context omitted.

It absolutely is deterministic, for any given seed value. Same seed = same output, every time, which is by definition deterministic.

only if temperature is 0, but are they truly determinstic? I thought transformer based llm's where not

temperature does not affect token prediction in the way you think. The seed value is still the seed value, before temperature calculations are performed. The randomness of an LLM is not related to its temperature. The seed value is what determines the output. For a specific seed value, say 42069, the LLM will always generate the same output, given the same input, given the same temperature.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#42

Post by tough »

mdaniel wrote: post #39
tough wrote: post #38

Earlier quoted context omitted.

People doesn't know github can manage secrets in its environment for CI? Antoher interesting fact is that most big vendors pay for gh to scan for leaked secrets and auto-revoke them if a public repo contains any (regex string matches sk-xxx thats one of the reasons why vendors use unique greppable starts of api keys with their ID.name on it

You're mistaking "know" with "care," since my experience has been that people know way more than they care And I'm pretty certain that private repos are exempt from the platform's built-in secret scanners because they, too, erroneously think no one can read them without an invitation. Turns out Duo was apparently just silently invited to every repo : - \

I also remember reading about how due to how the git backend works your private git repos branches could get exposed to the public, so yea don't treat a repository as a private password mananger

good point the scanner doesnt work on private repos =(

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#43

Post by tough »

33hsiidhkl wrote: post #41
tough wrote: post #35

Earlier quoted context omitted.

only if temperature is 0, but are they truly determinstic? I thought transformer based llm's where not

temperature does not affect token prediction in the way you think. The seed value is still the seed value, before temperature calculations are performed. The randomness of an LLM is not related to its temperature. The seed value is what determines the output. For a specific seed value, say 42069, the LLM will always generate the same output, given the same input, given the same temperature.

Thank you, I thought this wasn't the case (like it is with diffusion image models)

TIL

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#44

Post by margalabargala »

tough wrote: post #36
raphman wrote: post #33

Earlier quoted context omitted.

Why and how?

an agent does rm -rf / i think i saw it do it or try it and my computer shut down and restarted (mac) maybe it just deleted the project lol these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens. Honestly one should run only these on controlled env's like VM's or Docker. but YOLO amirite

That people allow these agents to just run arbitrary commands against their primary install is wild.

Part of this is the tool's fault. Anything like that should be done in a chroot.

Anything less is basically "twitch plays terminal" on your machine.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#45

Post by tough »

margalabargala wrote: post #44
tough wrote: post #36

Earlier quoted context omitted.

an agent does rm -rf / i think i saw it do it or try it and my computer shut down and restarted (mac) maybe it just deleted the project lol these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens. Honestly one should run only these on controlled env's like VM's or Docker. but YOLO amirite

That people allow these agents to just run arbitrary commands against their primary install is wild . Part of this is the tool's fault. Anything like that should be done in a chroot. Anything less is basically "twitch plays terminal" on your machine.

codex at least has limitations on what folders can operate.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#46

Post by ofjcihen »

tonyhart7 wrote: post #32

this is wild, how many security vuln that LLM can create where LLM dominate writing code???? I mean most coder is bad at security and we feed that into LLM so not surprise

This is what I’ve been telling people when they hand wave away concerns about LLM generated code security. The majority of what they were trained on was bare minimum security if anything.

You also can’t just fix it by saying “make it secure plz”.

If you don’t know enough to identify a security issue yourself you don’t know enough to know if the LLM caught them all.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#47

Post by throwaway314155 »

thaumasiotes wrote: post #40
zwnow wrote: post #25

Earlier quoted context omitted.

For tricking individuals your first got to contact them somehow. To trick an LLM you can just spam prompts.

You email them. It's called phishing.

Right and now there's a new vector for an old concept.

Re: Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft

#49

Post by TechDebtDevin »

tough wrote: post #36
raphman wrote: post #33

Earlier quoted context omitted.

Why and how?

an agent does rm -rf / i think i saw it do it or try it and my computer shut down and restarted (mac) maybe it just deleted the project lol these llms are really bad at keeping track of the real world, so they might think they're on the project folder but had just navigated back with cd to the user ~ root and so shit happens. Honestly one should run only these on controlled env's like VM's or Docker. but YOLO amirite

This is what happened. I was testing claude 4 and asked it to create a simple 1K LOC fyne android app. I have my repos stored outside of my linux user so the work it created was preserved. It essentially created a bash file that cd ~ && rm -rf / . All settings reset and documents/downloads disappeared lmfao. I don't ever really use my OS as primary storage, and any config or file of importance is backed up twice so it wasn't a big deal, but it was quite perplexing for a sec.
Post reply on HN