Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

noma.security

Post reply on HN
Print view View original

221–228 of 228 posts

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#221

Post by centuryfall »

brookst wrote: post #126
wzdd wrote: post #72

Earlier quoted context omitted.

The OP notes that they had to use special phrasing to get their exfil to work, so clearly GitHub was aware of the issue and made an attempt to prevent it. It seems like the proper fix is for GitHub not to allow their agentic workflow to execute in a public repo context if it also has private repo access. Or, to use your phrasing, for GitHub to flag and disallow this easily-detectable and dangerous type of misconfigur…

This “detectable and dangerous type of misconfiguration” is used by many developed daily and breaking it would break important workflows. It’s like saying that an OS should enforce that home directories can only have 0600 permissions. Yes, it prevents accidentally configuring world readable on files, but there are legit reasons for wanting to share a file from your home dir.

Why is that an issue though when it is the lesser of two evils? At the very minimum, regardless of “misconfiguration” or not, not having any type of warnings to advise against this behavior is quite bad.

Misconfiguration isn’t really the best word choice, either, because it’s definitely a both-sides problem.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#223

Post by centuryfall »

pkkm wrote: post #68

This reads like a marketing stunt for Noma. The cute name, the logo, the clickbait title, the dramatic tone in an article that seems targeted at a non-technical audience... And the actual vulnerability is what, that if you give an LLM private data and let random people interact with it, it may leak the data? Well, duh.

While it is definitely an issue if a single agent has access to both public and private data, this feature shouldn’t have been delivered where this is an allowed state. At least, GitHub should have ensured there are two kinds of agents: one for public, and one for internal, and prevent crossover between them. I get this doesn’t appear to be the most shiny feature, but the other, current side is just allowing Pandora’s box to be opened by naive policies.

Lastly, even with a private agent, being able to ask it for secrets and have it likely respond with them back is really, really, really bad.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#224

Post by partyficial »

voidUpdate wrote: post #35

'No Way to Prevent This,' Says Only Programming Concept Where This Regularly Happens

the current AI mania is trying to shoe-horn a creative system into a deterministic one.

LLMs are creative. Databases are deterministic.

There is no right or wrong in a 'zero money image'.

There is right and wrong in a 'zero money update'.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#225

Post by lukasco »

moron4hire wrote: post #134
lukasco wrote: post #96

Earlier quoted context omitted.

This is the real problem with LLMs. There is no way to separate code from data. At best, models could be trained on tokens that indicate untrusted data coming in. But then the untrusted tokens could also be messed with. I've wondered if it would be possible for there to be two input streams: 1, for prompt, 2 for untrusted data. But I suspect that transformers would still only optionally decide what each one was for.…

LLMs should never be trained on restricted data of any kind, as we have seen that they are able to reconstruct their training data. The idea that they could be trained on private/restricted/copyrighted data and that was ok because there wouldn't be redistributing that data should have been killed 3 years ago. Embedding vector indexes are how we separate code from data. Anything that is not for 100% unadulterated publ…

I feel like in some ways this problem is starting to self-correct, sadly by creating the dead internet. If there's no business model to creating content since it will get scraped, then no content will get created.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#227

Post by silon42 »

brookst wrote: post #121
coldtea wrote: post #67

Earlier quoted context omitted.

It's not about if it can happen or if it happens. It's about how easily it's mitigated completely. Use a proper db library which does escaping and it's completely eliminated.

Nit: modern DB libraries use wire protocols where SQL injection is mitigated by modeling parameters; it’s not just assembled to one big SQL statement and escaped. Agree with your point though. There will come a time when properly designed LLM apps are not vulnerable, and there will still be poorly designed apps that are.

Yes, if SQL statements would be replaced by a restricted object model, SQL injection is by definition impossible. But you can just "prompt inject" the LLM itself.
Post reply on HN